[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:38:41 dist-test-slave-z40b krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:38:41 dist-test-slave-z40b krb5kdc[399](info): set up 2 sockets
Sep 27 16:38:41 dist-test-slave-z40b krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 16:38:42.535617 392 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:38:42 dist-test-slave-z40b krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:38:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:38:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:38:42.646997 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36631
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:38:42.752372 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:38:42.752631 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:38:42.752697 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:38:42.756690 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:38:42.756764 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:38:42.756808 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:38:42.756857 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:38:42.756891 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:38:42.761672 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:38:42.762640 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:38:42.763695 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:38:42.771865 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:42.771865 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:42.771952 412 server_base.cc:1034] running on GCE node
W20260927 16:38:42.771847 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:42.772722 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:38:42.773667 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:38:42.774822 412 hybrid_clock.cc:648] HybridClock initialized: now 1790527122774762 us; error 72 us; skew 500 ppm
Sep 27 16:38:42 dist-test-slave-z40b krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:38:42.782823 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:38:42.783690 412 webserver.cc:460] Webserver started at http://127.0.98.62:38399/ using document root <none> and password file <none>
I20260927 16:38:42.784236 412 fs_manager.cc:359] Metadata directory not provided
I20260927 16:38:42.784309 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:38:42.784564 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:38:42.786549 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "f183639eba964836bd764a2baa6143bf"
format_stamp: "Formatted at 2026-09-27 16:38:42 on dist-test-slave-z40b"
server_key: "f1c6a2474754a60fbdd2cd9cd0109d30"
server_key_iv: "63ea0bcb54428105e6375066dff6b5a6"
server_key_version: "encryptionkey@0"
I20260927 16:38:42.787096 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "f183639eba964836bd764a2baa6143bf"
format_stamp: "Formatted at 2026-09-27 16:38:42 on dist-test-slave-z40b"
server_key: "f1c6a2474754a60fbdd2cd9cd0109d30"
server_key_iv: "63ea0bcb54428105e6375066dff6b5a6"
server_key_version: "encryptionkey@0"
I20260927 16:38:42.790604 412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 16:38:42.792870 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:42.794044 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 16:38:42.794159 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "f183639eba964836bd764a2baa6143bf"
format_stamp: "Formatted at 2026-09-27 16:38:42 on dist-test-slave-z40b"
server_key: "f1c6a2474754a60fbdd2cd9cd0109d30"
server_key_iv: "63ea0bcb54428105e6375066dff6b5a6"
server_key_version: "encryptionkey@0"
I20260927 16:38:42.794287 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:38:42.807200 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:38:42.809661 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:38:42.809860 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:38:42.816100 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36631
I20260927 16:38:42.816110 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36631 every 8 connection(s)
I20260927 16:38:42.816947 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:38:42.819448 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:42.822538 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 412
I20260927 16:38:42.822700 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:38:42.824240 392 external_mini_cluster.cc:1428] Setting key dbec886d6d7e8c2597f8e7b6fa3ab71a
I20260927 16:38:42.825785 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f183639eba964836bd764a2baa6143bf: Bootstrap starting.
I20260927 16:38:42.828851 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f183639eba964836bd764a2baa6143bf: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:42.829739 480 log.cc:824] T 00000000000000000000000000000000 P f183639eba964836bd764a2baa6143bf: Log is configured to *not* fsync() on all Append() calls
I20260927 16:38:42.831985 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f183639eba964836bd764a2baa6143bf: No bootstrap required, opened a new log
I20260927 16:38:42.836660 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P f183639eba964836bd764a2baa6143bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f183639eba964836bd764a2baa6143bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36631 } }
I20260927 16:38:42.836935 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P f183639eba964836bd764a2baa6143bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:42.836999 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P f183639eba964836bd764a2baa6143bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f183639eba964836bd764a2baa6143bf, State: Initialized, Role: FOLLOWER
I20260927 16:38:42.837455 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P f183639eba964836bd764a2baa6143bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f183639eba964836bd764a2baa6143bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36631 } }
I20260927 16:38:42.837607 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P f183639eba964836bd764a2baa6143bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:38:42.837673 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P f183639eba964836bd764a2baa6143bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:38:42.837771 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f183639eba964836bd764a2baa6143bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:42.838794 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P f183639eba964836bd764a2baa6143bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f183639eba964836bd764a2baa6143bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36631 } }
I20260927 16:38:42.839111 480 leader_election.cc:302] T 00000000000000000000000000000000 P f183639eba964836bd764a2baa6143bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f183639eba964836bd764a2baa6143bf; no voters:
I20260927 16:38:42.839418 480 leader_election.cc:288] T 00000000000000000000000000000000 P f183639eba964836bd764a2baa6143bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:38:42.839524 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f183639eba964836bd764a2baa6143bf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:38:42.839738 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P f183639eba964836bd764a2baa6143bf [term 1 LEADER]: Becoming Leader. State: Replica: f183639eba964836bd764a2baa6143bf, State: Running, Role: LEADER
I20260927 16:38:42.840044 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P f183639eba964836bd764a2baa6143bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f183639eba964836bd764a2baa6143bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36631 } }
I20260927 16:38:42.840559 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P f183639eba964836bd764a2baa6143bf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:38:42.841569 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P f183639eba964836bd764a2baa6143bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader f183639eba964836bd764a2baa6143bf. Latest consensus state: current_term: 1 leader_uuid: "f183639eba964836bd764a2baa6143bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f183639eba964836bd764a2baa6143bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36631 } } }
I20260927 16:38:42.841692 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P f183639eba964836bd764a2baa6143bf [sys.catalog]: This master's current role is: LEADER
I20260927 16:38:42.841995 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P f183639eba964836bd764a2baa6143bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f183639eba964836bd764a2baa6143bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f183639eba964836bd764a2baa6143bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36631 } } }
I20260927 16:38:42.842110 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P f183639eba964836bd764a2baa6143bf [sys.catalog]: This master's current role is: LEADER
I20260927 16:38:42.842495 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:38:42.845803 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 16:38:42 dist-test-slave-z40b krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:38:42.851116 494 catalog_manager.cc:1347] Generated new cluster ID: 449d9e309e064b35bd994a2a5bc9383f
I20260927 16:38:42.851265 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:38:42.853171 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:42.829278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45284 (local address 127.0.98.62:36631)
0927 16:38:42.829781 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:42.829803 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:42.830476 (+ 673us) server_negotiation.cc:407] Connection header received
0927 16:38:42.834473 (+ 3997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:42.834490 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:42.834771 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:42.835010 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:42.837984 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:42.839489 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:42.841360 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:42.842182 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:42.842410 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:42.847454 (+ 5044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:42.847460 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:42.847474 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:42.847505 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:42.849570 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:42.850101 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:42.850106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:42.850109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:42.850221 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:42.850557 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:42.850565 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:42.850566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:42.851037 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:42.851272 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:42.852058 (+ 786us) server_negotiation.cc:299] Negotiation successful
0927 16:38:42.852328 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":131,"threads_started":1}
I20260927 16:38:42.859905 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:38:42.860814 494 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:38:42.870188 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f183639eba964836bd764a2baa6143bf: Generated new TSK 0
I20260927 16:38:42.870939 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:38:42.901675 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36631
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:38:43.006423 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:38:43.006644 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:38:43.006685 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:38:43.010447 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:38:43.010499 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:38:43.010596 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:38:43.016069 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36631
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:38:43.016992 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:38:43.017915 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:38:43.026614 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:43.026744 506 server_base.cc:1034] running on GCE node
W20260927 16:38:43.026604 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:43.026600 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:43.027364 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:38:43.027941 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:38:43.029117 506 hybrid_clock.cc:648] HybridClock initialized: now 1790527123029107 us; error 36 us; skew 500 ppm
Sep 27 16:38:43 dist-test-slave-z40b krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:38:43.031711 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:38:43.032418 506 webserver.cc:460] Webserver started at http://127.0.98.1:37287/ using document root <none> and password file <none>
I20260927 16:38:43.032907 506 fs_manager.cc:359] Metadata directory not provided
I20260927 16:38:43.032984 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:38:43.033155 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:38:43.034878 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-0/data/instance:
uuid: "029215f2aa5d455dbf8cf2465875e5f7"
format_stamp: "Formatted at 2026-09-27 16:38:43 on dist-test-slave-z40b"
server_key: "91dcb5a661fbf6e6f4833341a45709c4"
server_key_iv: "1e5fe7c86964b182078bce3587ab3963"
server_key_version: "encryptionkey@0"
I20260927 16:38:43.035452 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-0/wal/instance:
uuid: "029215f2aa5d455dbf8cf2465875e5f7"
format_stamp: "Formatted at 2026-09-27 16:38:43 on dist-test-slave-z40b"
server_key: "91dcb5a661fbf6e6f4833341a45709c4"
server_key_iv: "1e5fe7c86964b182078bce3587ab3963"
server_key_version: "encryptionkey@0"
I20260927 16:38:43.038679 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260927 16:38:43.041041 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:43.042173 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:38:43.042299 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "029215f2aa5d455dbf8cf2465875e5f7"
format_stamp: "Formatted at 2026-09-27 16:38:43 on dist-test-slave-z40b"
server_key: "91dcb5a661fbf6e6f4833341a45709c4"
server_key_iv: "1e5fe7c86964b182078bce3587ab3963"
server_key_version: "encryptionkey@0"
I20260927 16:38:43.042397 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:38:43.069659 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:38:43.072281 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:38:43.072467 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:38:43.072974 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:38:43.073731 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:38:43.073800 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:43.073869 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:38:43.073912 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:43.082563 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36891
I20260927 16:38:43.082585 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36891 every 8 connection(s)
I20260927 16:38:43.083427 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:38:43.087152 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 506
I20260927 16:38:43.087284 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:38:43.087566 392 external_mini_cluster.cc:1428] Setting key bbf69f8c4bd1dcccdea9196b8e7d23ee
Sep 27 16:38:43 dist-test-slave-z40b krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:38:43.097705 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:43.085260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45641 (local address 127.0.98.62:36631)
0927 16:38:43.085386 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:43.085390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:43.086144 (+ 754us) server_negotiation.cc:407] Connection header received
0927 16:38:43.087622 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:43.087626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:43.087683 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:43.087761 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:43.089188 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:43.089689 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:43.091192 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:43.091445 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:43.091541 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:43.093923 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:43.093943 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:43.093946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:43.093965 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:43.095581 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:43.096107 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:43.096111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:43.096113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:43.096237 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:43.096607 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:43.096611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:43.096613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:43.096787 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:43.096955 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:43.097424 (+ 469us) server_negotiation.cc:299] Negotiation successful
0927 16:38:43.097539 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:38:43.098642 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:43.085601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36631 (local address 127.0.98.1:45641)
0927 16:38:43.086016 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:43.086052 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:43.087419 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:43.087964 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:43.087972 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:43.088444 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:43.089048 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:43.089064 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:43.089801 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:43.089804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:43.091073 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:43.091083 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:43.091631 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:43.091634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:43.091861 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:43.092441 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:43.092456 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:43.093777 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:43.095720 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:43.095726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:43.095737 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:43.095968 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:43.096330 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:43.096333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:43.096336 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:43.096514 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:43.096897 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:43.096903 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:43.097173 (+ 270us) client_negotiation.cc:769] Sending connection context
0927 16:38:43.097354 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 16:38:43.097650 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 16:38:43.099773 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36631
I20260927 16:38:43.100030 635 heartbeater.cc:464] Registering TS with master...
I20260927 16:38:43.100536 635 heartbeater.cc:511] Master 127.0.98.62:36631 requested a full tablet report, sending...
I20260927 16:38:43.101981 444 ts_manager.cc:194] Registered new tserver with Master: 029215f2aa5d455dbf8cf2465875e5f7 (127.0.98.1:36891)
I20260927 16:38:43.103026 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:38:43.113819 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36631
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:38:43.217417 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:38:43.217643 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:38:43.217684 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:38:43.221524 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:38:43.221576 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:38:43.221630 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:38:43.226648 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36631
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:38:43.227658 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:38:43.228636 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:38:43.237411 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:43.237426 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:43.237399 640 server_base.cc:1034] running on GCE node
W20260927 16:38:43.237351 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:43.238003 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:38:43.238611 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:38:43.239768 640 hybrid_clock.cc:648] HybridClock initialized: now 1790527123239754 us; error 24 us; skew 500 ppm
Sep 27 16:38:43 dist-test-slave-z40b krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:38:43.242554 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:38:43.243435 640 webserver.cc:460] Webserver started at http://127.0.98.2:39657/ using document root <none> and password file <none>
I20260927 16:38:43.243955 640 fs_manager.cc:359] Metadata directory not provided
I20260927 16:38:43.244040 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:38:43.244254 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:38:43.246294 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-1/data/instance:
uuid: "6897277fcee3492f9e4e4a84add4ebec"
format_stamp: "Formatted at 2026-09-27 16:38:43 on dist-test-slave-z40b"
server_key: "5b6a04eba868b6e2d4490013311eeed0"
server_key_iv: "e548945fd5c51923802caeecc9566bcc"
server_key_version: "encryptionkey@0"
I20260927 16:38:43.246853 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6897277fcee3492f9e4e4a84add4ebec"
format_stamp: "Formatted at 2026-09-27 16:38:43 on dist-test-slave-z40b"
server_key: "5b6a04eba868b6e2d4490013311eeed0"
server_key_iv: "e548945fd5c51923802caeecc9566bcc"
server_key_version: "encryptionkey@0"
I20260927 16:38:43.250514 640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 16:38:43.252902 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:43.254074 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:38:43.254215 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "6897277fcee3492f9e4e4a84add4ebec"
format_stamp: "Formatted at 2026-09-27 16:38:43 on dist-test-slave-z40b"
server_key: "5b6a04eba868b6e2d4490013311eeed0"
server_key_iv: "e548945fd5c51923802caeecc9566bcc"
server_key_version: "encryptionkey@0"
I20260927 16:38:43.254323 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:38:43.265683 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:38:43.268800 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:38:43.268989 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:38:43.269529 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:38:43.270278 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:38:43.270358 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:43.270427 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:38:43.270467 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:43.280507 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41091
I20260927 16:38:43.280529 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41091 every 8 connection(s)
I20260927 16:38:43.281337 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:38:43.289095 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 640
I20260927 16:38:43.289215 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:38:43.289469 392 external_mini_cluster.cc:1428] Setting key 71402ec182429cc8fe632a391b34c4fa
Sep 27 16:38:43 dist-test-slave-z40b krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:38:43.294857 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:43.283071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56555 (local address 127.0.98.62:36631)
0927 16:38:43.283200 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:43.283234 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:43.284036 (+ 802us) server_negotiation.cc:407] Connection header received
0927 16:38:43.285335 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:43.285338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:43.285392 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:43.285496 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:43.286568 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:43.287097 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:43.288220 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:43.288443 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:43.288563 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:43.291092 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:43.291095 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:43.291097 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:43.291117 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:43.292675 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:43.293264 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:43.293266 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:43.293267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:43.293333 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:43.293719 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:43.293725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:43.293730 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:43.293970 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:43.294073 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:43.294510 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 16:38:43.294657 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:38:43.295595 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:43.283493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36631 (local address 127.0.98.2:56555)
0927 16:38:43.283903 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:43.283942 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:43.285136 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:43.285655 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:43.285661 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:43.285996 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:43.286433 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:43.286445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:43.287239 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:43.287243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:43.288097 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:43.288105 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:43.288594 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:43.288598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:43.288776 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:43.289379 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:43.289393 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:43.290944 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:43.292784 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:43.292798 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:43.292812 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:43.293157 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:43.293421 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:43.293426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:43.293431 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:43.293624 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:43.294069 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:43.294073 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:43.294282 (+ 209us) client_negotiation.cc:769] Sending connection context
0927 16:38:43.294429 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 16:38:43.294669 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":103,"threads_started":1}
I20260927 16:38:43.296722 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36631
I20260927 16:38:43.296963 769 heartbeater.cc:464] Registering TS with master...
I20260927 16:38:43.297435 769 heartbeater.cc:511] Master 127.0.98.62:36631 requested a full tablet report, sending...
I20260927 16:38:43.298452 444 ts_manager.cc:194] Registered new tserver with Master: 6897277fcee3492f9e4e4a84add4ebec (127.0.98.2:41091)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 16:38:43.299244 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:38:43.316421 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36631
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:38:43.420977 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:38:43.421197 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:38:43.421236 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:38:43.425129 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:38:43.425176 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:38:43.425232 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:38:43.430119 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36631
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:38:43.430945 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:38:43.431866 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:38:43.440865 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:43.440907 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:43.440881 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:43.441130 774 server_base.cc:1034] running on GCE node
I20260927 16:38:43.441491 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:38:43.442097 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:38:43.443274 774 hybrid_clock.cc:648] HybridClock initialized: now 1790527123443253 us; error 40 us; skew 500 ppm
Sep 27 16:38:43 dist-test-slave-z40b krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:38:43.445904 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:38:43.446718 774 webserver.cc:460] Webserver started at http://127.0.98.3:42191/ using document root <none> and password file <none>
I20260927 16:38:43.447204 774 fs_manager.cc:359] Metadata directory not provided
I20260927 16:38:43.447316 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:38:43.447471 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:38:43.449247 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-2/data/instance:
uuid: "67a811a245184774b81187b48b589792"
format_stamp: "Formatted at 2026-09-27 16:38:43 on dist-test-slave-z40b"
server_key: "4694551af63a242cca83080b4b380c06"
server_key_iv: "63e37647c6248d9dc259aba6fc1ac901"
server_key_version: "encryptionkey@0"
I20260927 16:38:43.449693 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-2/wal/instance:
uuid: "67a811a245184774b81187b48b589792"
format_stamp: "Formatted at 2026-09-27 16:38:43 on dist-test-slave-z40b"
server_key: "4694551af63a242cca83080b4b380c06"
server_key_iv: "63e37647c6248d9dc259aba6fc1ac901"
server_key_version: "encryptionkey@0"
I20260927 16:38:43.453027 774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:38:43.455016 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:43.456112 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:38:43.456218 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "67a811a245184774b81187b48b589792"
format_stamp: "Formatted at 2026-09-27 16:38:43 on dist-test-slave-z40b"
server_key: "4694551af63a242cca83080b4b380c06"
server_key_iv: "63e37647c6248d9dc259aba6fc1ac901"
server_key_version: "encryptionkey@0"
I20260927 16:38:43.456290 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:38:43.464347 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:38:43.467299 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:38:43.467427 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:38:43.467856 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:38:43.468502 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:38:43.468544 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:43.468576 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:38:43.468590 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:43.477494 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36313
I20260927 16:38:43.477516 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36313 every 8 connection(s)
I20260927 16:38:43.478359 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:38:43.481760 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 774
I20260927 16:38:43.481874 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790527121515808-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:38:43.482155 392 external_mini_cluster.cc:1428] Setting key 6cbe7f30dc100e06e0a922216112262c
Sep 27 16:38:43 dist-test-slave-z40b krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:38:43.490547 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:43.479953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34185 (local address 127.0.98.62:36631)
0927 16:38:43.480096 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:43.480101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:43.480824 (+ 723us) server_negotiation.cc:407] Connection header received
0927 16:38:43.482094 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:43.482097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:43.482151 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:43.482241 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:43.483174 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:43.483883 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:43.484997 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:43.485221 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:43.485330 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:43.487467 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:43.487470 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:43.487473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:43.487502 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:43.488636 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:43.489084 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:43.489089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:43.489091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:43.489164 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:43.489493 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:43.489497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:43.489499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:43.489701 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:43.489819 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:43.490257 (+ 438us) server_negotiation.cc:299] Negotiation successful
0927 16:38:43.490363 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 16:38:43.491340 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:43.480284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36631 (local address 127.0.98.3:34185)
0927 16:38:43.480669 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:43.480725 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:43.481834 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:43.482388 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:43.482394 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:43.482673 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:43.483068 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:43.483079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:43.483993 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:43.483996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:43.484877 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:43.484885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:43.485358 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:43.485362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:43.485532 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:43.485975 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:43.485988 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:43.487324 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:43.488744 (+ 1420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:43.488749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:43.488760 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:43.488985 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:43.489249 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:43.489254 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:43.489255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:43.489385 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:43.489814 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:43.489819 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:43.490053 (+ 234us) client_negotiation.cc:769] Sending connection context
0927 16:38:43.490219 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 16:38:43.490516 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":97,"threads_started":1}
I20260927 16:38:43.492308 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36631
I20260927 16:38:43.492470 903 heartbeater.cc:464] Registering TS with master...
I20260927 16:38:43.492828 903 heartbeater.cc:511] Master 127.0.98.62:36631 requested a full tablet report, sending...
I20260927 16:38:43.493686 444 ts_manager.cc:194] Registered new tserver with Master: 67a811a245184774b81187b48b589792 (127.0.98.3:36313)
I20260927 16:38:43.494138 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34185
I20260927 16:38:43.495963 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 16:38:43 dist-test-slave-z40b krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527123, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 16:38:43 dist-test-slave-z40b krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527123, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 16:38:43.519974 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:43.511036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54394 (local address 127.0.98.1:36891)
0927 16:38:43.511397 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:43.511405 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:43.511426 (+ 21us) server_negotiation.cc:407] Connection header received
0927 16:38:43.511470 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:43.511474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:43.511633 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:43.511743 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:43.512319 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:43.513019 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:43.513741 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:43.514048 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:43.514118 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:43.516292 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:43.516302 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:43.516313 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:43.516334 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:43.518365 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:43.518752 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:43.518756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:43.518757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:43.518837 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:43.519049 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:43.519051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:43.519052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:43.519337 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:43.519478 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:43.519621 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 16:38:43.519762 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":156,"threads_started":1}
W20260927 16:38:43.520854 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54394
Sep 27 16:38:43 dist-test-slave-z40b krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 16:38:43 dist-test-slave-z40b krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 16:38:43.544414 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:43.536561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54400 (local address 127.0.98.1:36891)
0927 16:38:43.536696 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:43.536701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:43.536822 (+ 121us) server_negotiation.cc:407] Connection header received
0927 16:38:43.536978 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:43.536981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:43.537027 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:43.537111 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:43.537764 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:43.538272 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:43.539115 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:43.539400 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:43.539516 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:43.541654 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:43.541658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:43.541661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:43.541690 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:43.542875 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:43.543277 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:43.543280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:43.543281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:43.543345 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:43.543631 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:43.543634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:43.543636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:43.543782 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:43.543920 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:43.544154 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 16:38:43.544262 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:38:43.545323 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 506
I20260927 16:38:43.550902 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 640
I20260927 16:38:43.558018 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 774
I20260927 16:38:43.562816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 412
2026-09-27T16:38:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2042 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:38:43 dist-test-slave-z40b krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:38:43 dist-test-slave-z40b krb5kdc[917](info): set up 2 sockets
Sep 27 16:38:43 dist-test-slave-z40b krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 16:38:44.562597 392 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:38:44 dist-test-slave-z40b krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:38:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:38:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:38:44.674736 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34329
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34329
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:38:44.777488 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:38:44.777736 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:38:44.777805 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:38:44.781685 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:38:44.781756 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:38:44.781791 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:38:44.781844 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:38:44.781881 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:38:44.786897 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:38:44.787811 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:38:44.788708 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:38:44.796191 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:44.796234 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:44.796245 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:44.796494 930 server_base.cc:1034] running on GCE node
I20260927 16:38:44.796998 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:38:44.797981 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:38:44.799167 930 hybrid_clock.cc:648] HybridClock initialized: now 1790527124799154 us; error 34 us; skew 500 ppm
Sep 27 16:38:44 dist-test-slave-z40b krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:38:44.801824 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:38:44.802591 930 webserver.cc:460] Webserver started at http://127.0.98.62:44117/ using document root <none> and password file <none>
I20260927 16:38:44.803056 930 fs_manager.cc:359] Metadata directory not provided
I20260927 16:38:44.803107 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:38:44.803301 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:38:44.805104 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "5f7ab49335bd4022abd7d0f924abf80f"
format_stamp: "Formatted at 2026-09-27 16:38:44 on dist-test-slave-z40b"
server_key: "3fa172931b8d3439bf4e62b31028f82d"
server_key_iv: "6eafab297394e002916851c63b5fc1c5"
server_key_version: "encryptionkey@0"
I20260927 16:38:44.805570 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "5f7ab49335bd4022abd7d0f924abf80f"
format_stamp: "Formatted at 2026-09-27 16:38:44 on dist-test-slave-z40b"
server_key: "3fa172931b8d3439bf4e62b31028f82d"
server_key_iv: "6eafab297394e002916851c63b5fc1c5"
server_key_version: "encryptionkey@0"
I20260927 16:38:44.808903 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:38:44.810957 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:44.812139 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:38:44.812264 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "5f7ab49335bd4022abd7d0f924abf80f"
format_stamp: "Formatted at 2026-09-27 16:38:44 on dist-test-slave-z40b"
server_key: "3fa172931b8d3439bf4e62b31028f82d"
server_key_iv: "6eafab297394e002916851c63b5fc1c5"
server_key_version: "encryptionkey@0"
I20260927 16:38:44.812358 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:38:44.825495 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:38:44.828588 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:38:44.828719 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:38:44.834918 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34329
I20260927 16:38:44.834941 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34329 every 8 connection(s)
I20260927 16:38:44.835810 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:38:44.838483 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:44.839372 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 930
I20260927 16:38:44.839479 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:38:44.839715 392 external_mini_cluster.cc:1428] Setting key 158b58b931a71e13956448993a02d207
I20260927 16:38:44.845108 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f7ab49335bd4022abd7d0f924abf80f: Bootstrap starting.
I20260927 16:38:44.847522 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f7ab49335bd4022abd7d0f924abf80f: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:44.848217 998 log.cc:824] T 00000000000000000000000000000000 P 5f7ab49335bd4022abd7d0f924abf80f: Log is configured to *not* fsync() on all Append() calls
Sep 27 16:38:44 dist-test-slave-z40b krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:38:44.850047 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f7ab49335bd4022abd7d0f924abf80f: No bootstrap required, opened a new log
I20260927 16:38:44.853754 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:44.841039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44658 (local address 127.0.98.62:34329)
0927 16:38:44.841466 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:44.841492 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:44.841535 (+ 43us) server_negotiation.cc:407] Connection header received
0927 16:38:44.842331 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:44.842352 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:44.842753 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:44.843096 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:44.843757 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:44.845190 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:44.846002 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:44.846675 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:44.846835 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:44.849402 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:44.849406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:44.849418 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:44.849440 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:44.851008 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:44.851474 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:44.851479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:44.851483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:44.851581 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:44.851867 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:44.851869 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:44.851870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:44.852176 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:44.852306 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:44.852555 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 16:38:44.852830 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":91,"threads_started":1}
I20260927 16:38:44.853969 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f7ab49335bd4022abd7d0f924abf80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7ab49335bd4022abd7d0f924abf80f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } }
I20260927 16:38:44.854264 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f7ab49335bd4022abd7d0f924abf80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:44.854331 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f7ab49335bd4022abd7d0f924abf80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f7ab49335bd4022abd7d0f924abf80f, State: Initialized, Role: FOLLOWER
I20260927 16:38:44.854789 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f7ab49335bd4022abd7d0f924abf80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7ab49335bd4022abd7d0f924abf80f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } }
I20260927 16:38:44.854947 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f7ab49335bd4022abd7d0f924abf80f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:38:44.855016 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f7ab49335bd4022abd7d0f924abf80f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:38:44.855109 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f7ab49335bd4022abd7d0f924abf80f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:44.856114 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f7ab49335bd4022abd7d0f924abf80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7ab49335bd4022abd7d0f924abf80f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } }
I20260927 16:38:44.856434 998 leader_election.cc:302] T 00000000000000000000000000000000 P 5f7ab49335bd4022abd7d0f924abf80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f7ab49335bd4022abd7d0f924abf80f; no voters:
I20260927 16:38:44.856737 998 leader_election.cc:288] T 00000000000000000000000000000000 P 5f7ab49335bd4022abd7d0f924abf80f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:38:44.856879 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f7ab49335bd4022abd7d0f924abf80f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:38:44.857107 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f7ab49335bd4022abd7d0f924abf80f [term 1 LEADER]: Becoming Leader. State: Replica: 5f7ab49335bd4022abd7d0f924abf80f, State: Running, Role: LEADER
I20260927 16:38:44.857455 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f7ab49335bd4022abd7d0f924abf80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7ab49335bd4022abd7d0f924abf80f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } }
I20260927 16:38:44.857859 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f7ab49335bd4022abd7d0f924abf80f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:38:44.858893 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f7ab49335bd4022abd7d0f924abf80f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f7ab49335bd4022abd7d0f924abf80f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7ab49335bd4022abd7d0f924abf80f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } }
I20260927 16:38:44.859072 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f7ab49335bd4022abd7d0f924abf80f [sys.catalog]: This master's current role is: LEADER
I20260927 16:38:44.859338 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f7ab49335bd4022abd7d0f924abf80f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f7ab49335bd4022abd7d0f924abf80f. Latest consensus state: current_term: 1 leader_uuid: "5f7ab49335bd4022abd7d0f924abf80f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7ab49335bd4022abd7d0f924abf80f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } }
I20260927 16:38:44.859445 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f7ab49335bd4022abd7d0f924abf80f [sys.catalog]: This master's current role is: LEADER
I20260927 16:38:44.860170 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:38:44.862608 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:38:44.868495 1012 catalog_manager.cc:1347] Generated new cluster ID: 8ef9f502ccde45cab91cc3591e13e50d
I20260927 16:38:44.868602 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:38:44.874785 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:38:44.875713 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:38:44.884272 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f7ab49335bd4022abd7d0f924abf80f: Generated new TSK 0
I20260927 16:38:44.884856 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:38:44.913533 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34329
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:38:45.018492 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:38:45.018702 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:38:45.018741 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:38:45.022548 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:38:45.022601 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:38:45.022708 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:38:45.028678 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34329
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:38:45.029758 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:38:45.030699 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:38:45.041326 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:45.041368 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:45.041373 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:45.041808 1024 server_base.cc:1034] running on GCE node
I20260927 16:38:45.042219 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:38:45.042960 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:38:45.044113 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790527125044096 us; error 31 us; skew 500 ppm
Sep 27 16:38:45 dist-test-slave-z40b krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:38:45.046731 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:38:45.047577 1024 webserver.cc:460] Webserver started at http://127.0.98.1:38313/ using document root <none> and password file <none>
I20260927 16:38:45.048105 1024 fs_manager.cc:359] Metadata directory not provided
I20260927 16:38:45.048161 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:38:45.048377 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:38:45.050195 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-0/data/instance:
uuid: "176ca39183a54bbca329f4b9ffd810ac"
format_stamp: "Formatted at 2026-09-27 16:38:45 on dist-test-slave-z40b"
server_key: "462fd684d64fa66713c004615beebec8"
server_key_iv: "d29b229d43fa3e98b5a332c9e1bd8ae4"
server_key_version: "encryptionkey@0"
I20260927 16:38:45.050721 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-0/wal/instance:
uuid: "176ca39183a54bbca329f4b9ffd810ac"
format_stamp: "Formatted at 2026-09-27 16:38:45 on dist-test-slave-z40b"
server_key: "462fd684d64fa66713c004615beebec8"
server_key_iv: "d29b229d43fa3e98b5a332c9e1bd8ae4"
server_key_version: "encryptionkey@0"
I20260927 16:38:45.054222 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:38:45.056564 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:45.057678 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:38:45.057811 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "176ca39183a54bbca329f4b9ffd810ac"
format_stamp: "Formatted at 2026-09-27 16:38:45 on dist-test-slave-z40b"
server_key: "462fd684d64fa66713c004615beebec8"
server_key_iv: "d29b229d43fa3e98b5a332c9e1bd8ae4"
server_key_version: "encryptionkey@0"
I20260927 16:38:45.057912 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:38:45.069090 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:38:45.071943 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:38:45.072117 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:38:45.072616 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:38:45.073277 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:38:45.073339 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:45.073398 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:38:45.073451 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:45.081950 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33839
I20260927 16:38:45.081972 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33839 every 8 connection(s)
I20260927 16:38:45.082712 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:38:45.088709 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 1024
I20260927 16:38:45.088804 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:38:45.089056 392 external_mini_cluster.cc:1428] Setting key 6c05fcaefc658c4d39ea2e4b71c494e2
Sep 27 16:38:45 dist-test-slave-z40b krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:38:45.095762 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:45.084266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50909 (local address 127.0.98.62:34329)
0927 16:38:45.084380 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:45.084384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:45.085139 (+ 755us) server_negotiation.cc:407] Connection header received
0927 16:38:45.086431 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:45.086434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:45.086485 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:45.086560 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:45.087671 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.088206 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.089659 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.089952 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.090063 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:45.092430 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:45.092434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:45.092435 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:45.092457 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:45.093781 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:45.094298 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:45.094302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:45.094304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:45.094383 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:45.094697 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:45.094702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:45.094704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:45.094875 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:45.094977 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:45.095473 (+ 496us) server_negotiation.cc:299] Negotiation successful
0927 16:38:45.095601 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 16:38:45.096449 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:45.084592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34329 (local address 127.0.98.1:50909)
0927 16:38:45.085008 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:45.085047 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:45.086239 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:45.086706 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:45.086713 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:45.087015 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:45.087526 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:45.087540 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.088337 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.088342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:45.089541 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:45.089550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.090095 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.090098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:45.090324 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:45.090863 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:45.090878 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:45.092236 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:45.093916 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:45.093921 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:45.093933 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:45.094198 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:45.094474 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:45.094478 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:45.094480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:45.094606 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:45.094975 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:45.094980 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:45.095196 (+ 216us) client_negotiation.cc:769] Sending connection context
0927 16:38:45.095435 (+ 239us) client_negotiation.cc:240] Negotiation successful
0927 16:38:45.095711 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":129,"threads_started":1}
I20260927 16:38:45.097568 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34329
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:38:45.097786 1153 heartbeater.cc:464] Registering TS with master...
I20260927 16:38:45.098245 1153 heartbeater.cc:511] Master 127.0.98.62:34329 requested a full tablet report, sending...
I20260927 16:38:45.099660 962 ts_manager.cc:194] Registered new tserver with Master: 176ca39183a54bbca329f4b9ffd810ac (127.0.98.1:33839)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 16:38:45.100852 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:38:45.114473 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34329
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:38:45.217242 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:38:45.217475 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:38:45.217516 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:38:45.221378 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:38:45.221432 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:38:45.221487 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:38:45.226456 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34329
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:38:45.227461 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:38:45.228437 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:38:45.237067 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:45.237064 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:45.237353 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:45.237289 1158 server_base.cc:1034] running on GCE node
I20260927 16:38:45.237864 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:38:45.238425 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:38:45.239579 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790527125239551 us; error 39 us; skew 500 ppm
Sep 27 16:38:45 dist-test-slave-z40b krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:38:45.242184 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:38:45.243006 1158 webserver.cc:460] Webserver started at http://127.0.98.2:46339/ using document root <none> and password file <none>
I20260927 16:38:45.243585 1158 fs_manager.cc:359] Metadata directory not provided
I20260927 16:38:45.243671 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:38:45.243885 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:38:45.245792 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-1/data/instance:
uuid: "b23ecbefafcd43078bc165abc86fb57c"
format_stamp: "Formatted at 2026-09-27 16:38:45 on dist-test-slave-z40b"
server_key: "9ec570d40b87a92742fb723b2ab9d243"
server_key_iv: "fa3fc613c78b5ebc497b2a0d0301bc2c"
server_key_version: "encryptionkey@0"
I20260927 16:38:45.246336 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b23ecbefafcd43078bc165abc86fb57c"
format_stamp: "Formatted at 2026-09-27 16:38:45 on dist-test-slave-z40b"
server_key: "9ec570d40b87a92742fb723b2ab9d243"
server_key_iv: "fa3fc613c78b5ebc497b2a0d0301bc2c"
server_key_version: "encryptionkey@0"
I20260927 16:38:45.249913 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 16:38:45.252133 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:45.253216 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:38:45.253335 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "b23ecbefafcd43078bc165abc86fb57c"
format_stamp: "Formatted at 2026-09-27 16:38:45 on dist-test-slave-z40b"
server_key: "9ec570d40b87a92742fb723b2ab9d243"
server_key_iv: "fa3fc613c78b5ebc497b2a0d0301bc2c"
server_key_version: "encryptionkey@0"
I20260927 16:38:45.253436 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:38:45.266517 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:38:45.269533 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:38:45.269716 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:38:45.270241 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:38:45.270998 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:38:45.271063 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:45.271126 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:38:45.271173 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:45.280337 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42505
I20260927 16:38:45.280360 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42505 every 8 connection(s)
I20260927 16:38:45.281147 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 16:38:45 dist-test-slave-z40b krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:38:45.289757 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 1158
I20260927 16:38:45.289899 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:38:45.290134 392 external_mini_cluster.cc:1428] Setting key b4ef5afe21ad830d68d158110093f869
I20260927 16:38:45.293737 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:45.282863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42779 (local address 127.0.98.62:34329)
0927 16:38:45.282985 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:45.282988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:45.283783 (+ 795us) server_negotiation.cc:407] Connection header received
0927 16:38:45.285008 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:45.285011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:45.285058 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:45.285131 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:45.286193 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.286703 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.287766 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.287983 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.288061 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:45.290357 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:45.290361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:45.290362 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:45.290391 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:45.291852 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:45.292299 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:45.292303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:45.292305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:45.292381 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:45.292718 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:45.292723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:45.292725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:45.292892 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:45.292993 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:45.293446 (+ 453us) server_negotiation.cc:299] Negotiation successful
0927 16:38:45.293558 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 16:38:45.294530 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:45.283170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34329 (local address 127.0.98.2:42779)
0927 16:38:45.283636 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:45.283676 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:45.284833 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:45.285270 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:45.285276 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:45.285603 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:45.286079 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:45.286092 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.286799 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.286801 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:45.287664 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:45.287673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.288101 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.288104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:45.288229 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:45.288710 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:45.288723 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:45.290184 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:45.291957 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:45.291962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:45.291974 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:45.292203 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:45.292485 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:45.292489 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:45.292491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:45.292627 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:45.292991 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:45.292997 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:45.293220 (+ 223us) client_negotiation.cc:769] Sending connection context
0927 16:38:45.293403 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 16:38:45.293709 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":104,"threads_started":1}
I20260927 16:38:45.295601 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34329
I20260927 16:38:45.295835 1287 heartbeater.cc:464] Registering TS with master...
I20260927 16:38:45.296406 1287 heartbeater.cc:511] Master 127.0.98.62:34329 requested a full tablet report, sending...
I20260927 16:38:45.297529 962 ts_manager.cc:194] Registered new tserver with Master: b23ecbefafcd43078bc165abc86fb57c (127.0.98.2:42505)
I20260927 16:38:45.298324 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42779
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:38:45.316015 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34329
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:38:45.420318 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:38:45.420583 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:38:45.420678 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:38:45.424558 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:38:45.424634 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:38:45.424721 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:38:45.429574 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34329
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:38:45.430442 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:38:45.431365 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:38:45.440258 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:45.440274 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:45.440313 1292 server_base.cc:1034] running on GCE node
W20260927 16:38:45.440267 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:45.440788 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:38:45.441376 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:38:45.442523 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790527125442503 us; error 36 us; skew 500 ppm
Sep 27 16:38:45 dist-test-slave-z40b krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:38:45.445048 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:38:45.445837 1292 webserver.cc:460] Webserver started at http://127.0.98.3:36449/ using document root <none> and password file <none>
I20260927 16:38:45.446388 1292 fs_manager.cc:359] Metadata directory not provided
I20260927 16:38:45.446471 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:38:45.446682 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:38:45.448576 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-2/data/instance:
uuid: "82dee4258f404d3da5970b7842aecf67"
format_stamp: "Formatted at 2026-09-27 16:38:45 on dist-test-slave-z40b"
server_key: "ff46ac13a823da5315e36e7925ea7b13"
server_key_iv: "bb73407b57703727b9594b495363daf7"
server_key_version: "encryptionkey@0"
I20260927 16:38:45.449084 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-2/wal/instance:
uuid: "82dee4258f404d3da5970b7842aecf67"
format_stamp: "Formatted at 2026-09-27 16:38:45 on dist-test-slave-z40b"
server_key: "ff46ac13a823da5315e36e7925ea7b13"
server_key_iv: "bb73407b57703727b9594b495363daf7"
server_key_version: "encryptionkey@0"
I20260927 16:38:45.452373 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:38:45.454370 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:45.455407 1292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:38:45.455514 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "82dee4258f404d3da5970b7842aecf67"
format_stamp: "Formatted at 2026-09-27 16:38:45 on dist-test-slave-z40b"
server_key: "ff46ac13a823da5315e36e7925ea7b13"
server_key_iv: "bb73407b57703727b9594b495363daf7"
server_key_version: "encryptionkey@0"
I20260927 16:38:45.455606 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:38:45.477797 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:38:45.480767 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:38:45.481005 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:38:45.481542 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:38:45.482396 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:38:45.482463 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:45.482538 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:38:45.482587 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:45.491230 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37371
I20260927 16:38:45.491281 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37371 every 8 connection(s)
I20260927 16:38:45.492022 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 16:38:45 dist-test-slave-z40b krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:38:45.501412 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 1292
I20260927 16:38:45.501518 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790527121515808-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:38:45.501760 392 external_mini_cluster.cc:1428] Setting key d56c86398209f0793fc944530fc05139
I20260927 16:38:45.504235 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:45.493674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39669 (local address 127.0.98.62:34329)
0927 16:38:45.493853 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:45.493857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:45.494527 (+ 670us) server_negotiation.cc:407] Connection header received
0927 16:38:45.495651 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:45.495654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:45.495696 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:45.495792 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:45.496783 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.497287 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.498317 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.498542 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.498621 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:45.500739 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:45.500742 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:45.500744 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:45.500770 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:45.502162 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:45.502715 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:45.502719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:45.502721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:45.502818 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:45.503225 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:45.503229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:45.503230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:45.503442 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:45.503538 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:45.504048 (+ 510us) server_negotiation.cc:299] Negotiation successful
0927 16:38:45.504132 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 16:38:45.505005 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:45.494020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34329 (local address 127.0.98.3:39669)
0927 16:38:45.494386 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:45.494420 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:45.495457 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:45.495897 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:45.495902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:45.496200 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:45.496610 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:45.496622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.497397 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.497399 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:45.498207 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:45.498215 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.498661 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.498663 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:45.498795 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:45.499332 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:45.499346 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:45.500592 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:45.502266 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:45.502270 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:45.502282 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:45.502602 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:45.502922 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:45.502927 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:45.502929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:45.503102 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:45.503540 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:45.503544 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:45.503838 (+ 294us) client_negotiation.cc:769] Sending connection context
0927 16:38:45.503990 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 16:38:45.504197 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":97,"threads_started":1}
I20260927 16:38:45.506049 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34329
I20260927 16:38:45.506283 1421 heartbeater.cc:464] Registering TS with master...
I20260927 16:38:45.506712 1421 heartbeater.cc:511] Master 127.0.98.62:34329 requested a full tablet report, sending...
I20260927 16:38:45.507679 962 ts_manager.cc:194] Registered new tserver with Master: 82dee4258f404d3da5970b7842aecf67 (127.0.98.3:37371)
I20260927 16:38:45.508255 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39669
I20260927 16:38:45.515456 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:38:45.525089 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:45.518517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43334 (local address 127.0.98.62:34329)
0927 16:38:45.518651 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:45.518657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:45.518759 (+ 102us) server_negotiation.cc:407] Connection header received
0927 16:38:45.518841 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:45.518843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:45.518876 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:45.518947 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:45.519596 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.520043 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.520748 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.520936 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.521006 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:45.522204 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:45.522207 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:45.522209 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:45.522226 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:45.523696 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:45.524055 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:45.524058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:45.524059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:45.524128 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:45.524344 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:45.524346 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:45.524347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:45.524527 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:45.524624 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:45.524808 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 16:38:45.524893 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:38:45.529222 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:38:45.532003 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:38:45.545471 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:45.540211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45620 (local address 127.0.98.2:42505)
0927 16:38:45.540780 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:45.540809 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:45.540830 (+ 21us) server_negotiation.cc:407] Connection header received
0927 16:38:45.540901 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:45.540907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:45.541088 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:45.541444 (+ 356us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:38:45.541599 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.542663 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.544238 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.544751 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.544931 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:45.545074 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:45.545213 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 16:38:45.545313 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":106,"threads_started":1}
I20260927 16:38:45.545478 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:45.539621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42505 (local address 127.0.0.1:45620)
0927 16:38:45.540243 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:45.540285 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:45.540420 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:45.541218 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:45.541220 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:45.541248 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:38:45.541479 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:45.541488 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.542793 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.542795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:45.544100 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:45.544109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.544927 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.544930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:45.545031 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:45.545064 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 16:38:45.545178 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 16:38:45.545352 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":215,"spinlock_wait_cycles":52480,"thread_start_us":101,"threads_started":1}
I20260927 16:38:45.546001 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:45.540172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51052 (local address 127.0.98.1:33839)
0927 16:38:45.540608 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:45.540620 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:45.541082 (+ 462us) server_negotiation.cc:407] Connection header received
0927 16:38:45.541188 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:45.541193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:45.541341 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:45.541489 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:38:45.541788 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.542438 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.544056 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.544886 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.544995 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:45.545143 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:45.545732 (+ 589us) server_negotiation.cc:299] Negotiation successful
0927 16:38:45.545827 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":118,"threads_started":1}
I20260927 16:38:45.546000 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:45.539585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37620 (local address 127.0.98.3:37371)
0927 16:38:45.540684 (+ 1099us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:45.540698 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:45.540719 (+ 21us) server_negotiation.cc:407] Connection header received
0927 16:38:45.540785 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:45.540790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:45.540981 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:45.541123 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:38:45.541553 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.542664 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.544216 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.544979 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.545063 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:45.545169 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:45.545766 (+ 597us) server_negotiation.cc:299] Negotiation successful
0927 16:38:45.545845 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":990,"thread_start_us":130,"threads_started":1}
I20260927 16:38:45.546339 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:45.539463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37371 (local address 127.0.0.1:37620)
0927 16:38:45.540250 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:45.540285 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:45.540420 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:45.541214 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:45.541219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:45.541247 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:38:45.541431 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:45.541437 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.542824 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.542827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:45.544100 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:45.544109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.545214 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.545217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:45.545442 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:45.545482 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 16:38:45.545542 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 16:38:45.546221 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":314,"spinlock_wait_cycles":7040,"thread_start_us":61,"threads_started":1}
I20260927 16:38:45.546506 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:45.539833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33839 (local address 127.0.0.1:51052)
0927 16:38:45.540999 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:45.541010 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:45.541087 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:45.541590 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:45.541593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:45.541610 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:38:45.541688 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:45.541692 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.542593 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.542597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:45.543941 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:45.543950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.545207 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.545211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:45.545425 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:45.545465 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 16:38:45.546139 (+ 674us) client_negotiation.cc:240] Negotiation successful
0927 16:38:45.546330 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1116,"mutex_wait_us":247,"thread_start_us":93,"threads_started":1}
I20260927 16:38:45.548326 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 1332d5ebe43e467bbe611eb92e0be5dd (DEFAULT_TABLE table=test-table [id=895ec4181b5b4206abeebdbb62cb36b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:38:45.548846 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 1332d5ebe43e467bbe611eb92e0be5dd (DEFAULT_TABLE table=test-table [id=895ec4181b5b4206abeebdbb62cb36b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:38:45.549261 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1332d5ebe43e467bbe611eb92e0be5dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:45.549669 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1332d5ebe43e467bbe611eb92e0be5dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:45.554661 1439 tablet_bootstrap.cc:492] T 1332d5ebe43e467bbe611eb92e0be5dd P 176ca39183a54bbca329f4b9ffd810ac: Bootstrap starting.
I20260927 16:38:45.554661 1438 tablet_bootstrap.cc:492] T 1332d5ebe43e467bbe611eb92e0be5dd P b23ecbefafcd43078bc165abc86fb57c: Bootstrap starting.
I20260927 16:38:45.556702 1439 tablet_bootstrap.cc:654] T 1332d5ebe43e467bbe611eb92e0be5dd P 176ca39183a54bbca329f4b9ffd810ac: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:45.556864 1438 tablet_bootstrap.cc:654] T 1332d5ebe43e467bbe611eb92e0be5dd P b23ecbefafcd43078bc165abc86fb57c: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:45.556960 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 1332d5ebe43e467bbe611eb92e0be5dd (DEFAULT_TABLE table=test-table [id=895ec4181b5b4206abeebdbb62cb36b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:38:45.557899 1439 log.cc:824] T 1332d5ebe43e467bbe611eb92e0be5dd P 176ca39183a54bbca329f4b9ffd810ac: Log is configured to *not* fsync() on all Append() calls
I20260927 16:38:45.558029 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1332d5ebe43e467bbe611eb92e0be5dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:45.558072 1438 log.cc:824] T 1332d5ebe43e467bbe611eb92e0be5dd P b23ecbefafcd43078bc165abc86fb57c: Log is configured to *not* fsync() on all Append() calls
I20260927 16:38:45.559818 1439 tablet_bootstrap.cc:492] T 1332d5ebe43e467bbe611eb92e0be5dd P 176ca39183a54bbca329f4b9ffd810ac: No bootstrap required, opened a new log
I20260927 16:38:45.559895 1438 tablet_bootstrap.cc:492] T 1332d5ebe43e467bbe611eb92e0be5dd P b23ecbefafcd43078bc165abc86fb57c: No bootstrap required, opened a new log
I20260927 16:38:45.560011 1439 ts_tablet_manager.cc:1397] T 1332d5ebe43e467bbe611eb92e0be5dd P 176ca39183a54bbca329f4b9ffd810ac: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:38:45.560039 1438 ts_tablet_manager.cc:1397] T 1332d5ebe43e467bbe611eb92e0be5dd P b23ecbefafcd43078bc165abc86fb57c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:38:45.562628 1439 raft_consensus.cc:348] T 1332d5ebe43e467bbe611eb92e0be5dd P 176ca39183a54bbca329f4b9ffd810ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dee4258f404d3da5970b7842aecf67" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37371 } } peers { permanent_uuid: "b23ecbefafcd43078bc165abc86fb57c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "176ca39183a54bbca329f4b9ffd810ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33839 } }
I20260927 16:38:45.562841 1439 raft_consensus.cc:374] T 1332d5ebe43e467bbe611eb92e0be5dd P 176ca39183a54bbca329f4b9ffd810ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:45.562907 1439 raft_consensus.cc:729] T 1332d5ebe43e467bbe611eb92e0be5dd P 176ca39183a54bbca329f4b9ffd810ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176ca39183a54bbca329f4b9ffd810ac, State: Initialized, Role: FOLLOWER
I20260927 16:38:45.563382 1439 consensus_queue.cc:260] T 1332d5ebe43e467bbe611eb92e0be5dd P 176ca39183a54bbca329f4b9ffd810ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dee4258f404d3da5970b7842aecf67" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37371 } } peers { permanent_uuid: "b23ecbefafcd43078bc165abc86fb57c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "176ca39183a54bbca329f4b9ffd810ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33839 } }
I20260927 16:38:45.563915 1153 heartbeater.cc:503] Master 127.0.98.62:34329 was elected leader, sending a full tablet report...
I20260927 16:38:45.563863 1438 raft_consensus.cc:348] T 1332d5ebe43e467bbe611eb92e0be5dd P b23ecbefafcd43078bc165abc86fb57c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dee4258f404d3da5970b7842aecf67" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37371 } } peers { permanent_uuid: "b23ecbefafcd43078bc165abc86fb57c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "176ca39183a54bbca329f4b9ffd810ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33839 } }
I20260927 16:38:45.564086 1438 raft_consensus.cc:374] T 1332d5ebe43e467bbe611eb92e0be5dd P b23ecbefafcd43078bc165abc86fb57c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:45.564144 1438 raft_consensus.cc:729] T 1332d5ebe43e467bbe611eb92e0be5dd P b23ecbefafcd43078bc165abc86fb57c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b23ecbefafcd43078bc165abc86fb57c, State: Initialized, Role: FOLLOWER
I20260927 16:38:45.564530 1438 consensus_queue.cc:260] T 1332d5ebe43e467bbe611eb92e0be5dd P b23ecbefafcd43078bc165abc86fb57c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dee4258f404d3da5970b7842aecf67" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37371 } } peers { permanent_uuid: "b23ecbefafcd43078bc165abc86fb57c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "176ca39183a54bbca329f4b9ffd810ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33839 } }
I20260927 16:38:45.565469 1438 ts_tablet_manager.cc:1428] T 1332d5ebe43e467bbe611eb92e0be5dd P b23ecbefafcd43078bc165abc86fb57c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 16:38:45.565558 1439 ts_tablet_manager.cc:1428] T 1332d5ebe43e467bbe611eb92e0be5dd P 176ca39183a54bbca329f4b9ffd810ac: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 16:38:45.565824 1287 heartbeater.cc:503] Master 127.0.98.62:34329 was elected leader, sending a full tablet report...
I20260927 16:38:45.566220 1444 tablet_bootstrap.cc:492] T 1332d5ebe43e467bbe611eb92e0be5dd P 82dee4258f404d3da5970b7842aecf67: Bootstrap starting.
I20260927 16:38:45.568290 1444 tablet_bootstrap.cc:654] T 1332d5ebe43e467bbe611eb92e0be5dd P 82dee4258f404d3da5970b7842aecf67: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:45.569343 1444 log.cc:824] T 1332d5ebe43e467bbe611eb92e0be5dd P 82dee4258f404d3da5970b7842aecf67: Log is configured to *not* fsync() on all Append() calls
I20260927 16:38:45.570770 1444 tablet_bootstrap.cc:492] T 1332d5ebe43e467bbe611eb92e0be5dd P 82dee4258f404d3da5970b7842aecf67: No bootstrap required, opened a new log
I20260927 16:38:45.570952 1444 ts_tablet_manager.cc:1397] T 1332d5ebe43e467bbe611eb92e0be5dd P 82dee4258f404d3da5970b7842aecf67: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:38:45.573417 1444 raft_consensus.cc:348] T 1332d5ebe43e467bbe611eb92e0be5dd P 82dee4258f404d3da5970b7842aecf67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dee4258f404d3da5970b7842aecf67" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37371 } } peers { permanent_uuid: "b23ecbefafcd43078bc165abc86fb57c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "176ca39183a54bbca329f4b9ffd810ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33839 } }
I20260927 16:38:45.573580 1444 raft_consensus.cc:374] T 1332d5ebe43e467bbe611eb92e0be5dd P 82dee4258f404d3da5970b7842aecf67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:45.573608 1444 raft_consensus.cc:729] T 1332d5ebe43e467bbe611eb92e0be5dd P 82dee4258f404d3da5970b7842aecf67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82dee4258f404d3da5970b7842aecf67, State: Initialized, Role: FOLLOWER
I20260927 16:38:45.573917 1444 consensus_queue.cc:260] T 1332d5ebe43e467bbe611eb92e0be5dd P 82dee4258f404d3da5970b7842aecf67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dee4258f404d3da5970b7842aecf67" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37371 } } peers { permanent_uuid: "b23ecbefafcd43078bc165abc86fb57c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "176ca39183a54bbca329f4b9ffd810ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33839 } }
I20260927 16:38:45.574410 1421 heartbeater.cc:503] Master 127.0.98.62:34329 was elected leader, sending a full tablet report...
I20260927 16:38:45.574647 1444 ts_tablet_manager.cc:1428] T 1332d5ebe43e467bbe611eb92e0be5dd P 82dee4258f404d3da5970b7842aecf67: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260927 16:38:45.583747 1154 tablet.cc:2369] T 1332d5ebe43e467bbe611eb92e0be5dd P 176ca39183a54bbca329f4b9ffd810ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:38:45.742901 1422 tablet.cc:2369] T 1332d5ebe43e467bbe611eb92e0be5dd P 82dee4258f404d3da5970b7842aecf67: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:38:45.782338 1288 tablet.cc:2369] T 1332d5ebe43e467bbe611eb92e0be5dd P b23ecbefafcd43078bc165abc86fb57c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:38:45.839807 1443 raft_consensus.cc:482] T 1332d5ebe43e467bbe611eb92e0be5dd P b23ecbefafcd43078bc165abc86fb57c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:38:45.839993 1443 raft_consensus.cc:504] T 1332d5ebe43e467bbe611eb92e0be5dd P b23ecbefafcd43078bc165abc86fb57c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dee4258f404d3da5970b7842aecf67" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37371 } } peers { permanent_uuid: "b23ecbefafcd43078bc165abc86fb57c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "176ca39183a54bbca329f4b9ffd810ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33839 } }
I20260927 16:38:45.841065 1443 leader_election.cc:288] T 1332d5ebe43e467bbe611eb92e0be5dd P b23ecbefafcd43078bc165abc86fb57c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82dee4258f404d3da5970b7842aecf67 (127.0.98.3:37371), 176ca39183a54bbca329f4b9ffd810ac (127.0.98.1:33839)
I20260927 16:38:45.846421 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:45.841507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35747 (local address 127.0.98.1:33839)
0927 16:38:45.841627 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:45.841632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:45.841870 (+ 238us) server_negotiation.cc:407] Connection header received
0927 16:38:45.842005 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:45.842008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:45.842050 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:45.842139 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:38:45.842557 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.843258 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.844801 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.845541 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.845636 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:45.845733 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:45.846145 (+ 412us) server_negotiation.cc:299] Negotiation successful
0927 16:38:45.846240 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 16:38:45.846483 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:45.841425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46219 (local address 127.0.98.3:37371)
0927 16:38:45.841666 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:45.841672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:45.841749 (+ 77us) server_negotiation.cc:407] Connection header received
0927 16:38:45.841947 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:45.841953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:45.842014 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:45.842100 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:38:45.842407 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.843238 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.844908 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.845714 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.845818 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:45.845890 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:45.846309 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 16:38:45.846375 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260927 16:38:45.846504 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:45.841321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37371 (local address 127.0.98.2:46219)
0927 16:38:45.841628 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:45.841647 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:45.841779 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:45.842132 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:45.842135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:45.842153 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:38:45.842292 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:45.842297 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.843348 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.843355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:45.844714 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:45.844728 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.846001 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.846018 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:45.846158 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:45.846189 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 16:38:45.846291 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 16:38:45.846358 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":1,"thread_start_us":96,"threads_started":1}
I20260927 16:38:45.846894 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:45.841321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33839 (local address 127.0.98.2:35747)
0927 16:38:45.841776 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:45.841788 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:45.841878 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:45.842245 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:45.842250 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:45.842273 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:38:45.842432 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:45.842441 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.843374 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.843377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:45.844627 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:45.844642 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.845847 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.845852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:45.845981 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:45.846027 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 16:38:45.846606 (+ 579us) client_negotiation.cc:240] Negotiation successful
0927 16:38:45.846696 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":46,"thread_start_us":78,"threads_started":1}
I20260927 16:38:45.847291 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1332d5ebe43e467bbe611eb92e0be5dd" candidate_uuid: "b23ecbefafcd43078bc165abc86fb57c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82dee4258f404d3da5970b7842aecf67" is_pre_election: true
I20260927 16:38:45.847517 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1332d5ebe43e467bbe611eb92e0be5dd" candidate_uuid: "b23ecbefafcd43078bc165abc86fb57c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176ca39183a54bbca329f4b9ffd810ac" is_pre_election: true
I20260927 16:38:45.847604 1375 raft_consensus.cc:2454] T 1332d5ebe43e467bbe611eb92e0be5dd P 82dee4258f404d3da5970b7842aecf67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b23ecbefafcd43078bc165abc86fb57c in term 0.
I20260927 16:38:45.847792 1107 raft_consensus.cc:2454] T 1332d5ebe43e467bbe611eb92e0be5dd P 176ca39183a54bbca329f4b9ffd810ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b23ecbefafcd43078bc165abc86fb57c in term 0.
I20260927 16:38:45.848119 1176 leader_election.cc:302] T 1332d5ebe43e467bbe611eb92e0be5dd P b23ecbefafcd43078bc165abc86fb57c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82dee4258f404d3da5970b7842aecf67, b23ecbefafcd43078bc165abc86fb57c; no voters:
I20260927 16:38:45.848407 1443 raft_consensus.cc:2781] T 1332d5ebe43e467bbe611eb92e0be5dd P b23ecbefafcd43078bc165abc86fb57c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:38:45.848538 1443 raft_consensus.cc:482] T 1332d5ebe43e467bbe611eb92e0be5dd P b23ecbefafcd43078bc165abc86fb57c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:38:45.848604 1443 raft_consensus.cc:3037] T 1332d5ebe43e467bbe611eb92e0be5dd P b23ecbefafcd43078bc165abc86fb57c [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:45.850313 1443 raft_consensus.cc:504] T 1332d5ebe43e467bbe611eb92e0be5dd P b23ecbefafcd43078bc165abc86fb57c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dee4258f404d3da5970b7842aecf67" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37371 } } peers { permanent_uuid: "b23ecbefafcd43078bc165abc86fb57c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "176ca39183a54bbca329f4b9ffd810ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33839 } }
I20260927 16:38:45.850647 1443 leader_election.cc:288] T 1332d5ebe43e467bbe611eb92e0be5dd P b23ecbefafcd43078bc165abc86fb57c [CANDIDATE]: Term 1 election: Requested vote from peers 82dee4258f404d3da5970b7842aecf67 (127.0.98.3:37371), 176ca39183a54bbca329f4b9ffd810ac (127.0.98.1:33839)
I20260927 16:38:45.850955 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1332d5ebe43e467bbe611eb92e0be5dd" candidate_uuid: "b23ecbefafcd43078bc165abc86fb57c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176ca39183a54bbca329f4b9ffd810ac"
I20260927 16:38:45.851091 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1332d5ebe43e467bbe611eb92e0be5dd" candidate_uuid: "b23ecbefafcd43078bc165abc86fb57c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82dee4258f404d3da5970b7842aecf67"
I20260927 16:38:45.851104 1107 raft_consensus.cc:3037] T 1332d5ebe43e467bbe611eb92e0be5dd P 176ca39183a54bbca329f4b9ffd810ac [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:45.851230 1375 raft_consensus.cc:3037] T 1332d5ebe43e467bbe611eb92e0be5dd P 82dee4258f404d3da5970b7842aecf67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:45.852391 1375 raft_consensus.cc:2454] T 1332d5ebe43e467bbe611eb92e0be5dd P 82dee4258f404d3da5970b7842aecf67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b23ecbefafcd43078bc165abc86fb57c in term 1.
I20260927 16:38:45.852389 1107 raft_consensus.cc:2454] T 1332d5ebe43e467bbe611eb92e0be5dd P 176ca39183a54bbca329f4b9ffd810ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b23ecbefafcd43078bc165abc86fb57c in term 1.
I20260927 16:38:45.852694 1175 leader_election.cc:302] T 1332d5ebe43e467bbe611eb92e0be5dd P b23ecbefafcd43078bc165abc86fb57c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176ca39183a54bbca329f4b9ffd810ac, b23ecbefafcd43078bc165abc86fb57c; no voters:
I20260927 16:38:45.852942 1443 raft_consensus.cc:2781] T 1332d5ebe43e467bbe611eb92e0be5dd P b23ecbefafcd43078bc165abc86fb57c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:38:45.853107 1443 raft_consensus.cc:686] T 1332d5ebe43e467bbe611eb92e0be5dd P b23ecbefafcd43078bc165abc86fb57c [term 1 LEADER]: Becoming Leader. State: Replica: b23ecbefafcd43078bc165abc86fb57c, State: Running, Role: LEADER
I20260927 16:38:45.853435 1443 consensus_queue.cc:237] T 1332d5ebe43e467bbe611eb92e0be5dd P b23ecbefafcd43078bc165abc86fb57c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dee4258f404d3da5970b7842aecf67" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37371 } } peers { permanent_uuid: "b23ecbefafcd43078bc165abc86fb57c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "176ca39183a54bbca329f4b9ffd810ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33839 } }
I20260927 16:38:45.856329 962 catalog_manager.cc:5495] T 1332d5ebe43e467bbe611eb92e0be5dd P b23ecbefafcd43078bc165abc86fb57c reported cstate change: term changed from 0 to 1, leader changed from <none> to b23ecbefafcd43078bc165abc86fb57c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b23ecbefafcd43078bc165abc86fb57c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dee4258f404d3da5970b7842aecf67" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b23ecbefafcd43078bc165abc86fb57c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "176ca39183a54bbca329f4b9ffd810ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33839 } health_report { overall_health: UNKNOWN } } }
Sep 27 16:38:45 dist-test-slave-z40b krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 16:38:45.909325 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:45.901075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51058 (local address 127.0.98.1:33839)
0927 16:38:45.901225 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:45.901229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:45.901385 (+ 156us) server_negotiation.cc:407] Connection header received
0927 16:38:45.901467 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:45.901469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:45.901509 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:45.901575 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:45.902214 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.902713 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.903505 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.903704 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.903771 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:45.906162 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:45.906172 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:45.906186 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:45.906205 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:45.907650 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:45.908081 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:45.908087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:45.908088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:45.908169 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:45.908446 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:45.908450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:45.908451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:45.908694 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:45.908901 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:45.909059 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 16:38:45.909180 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 16:38:45 dist-test-slave-z40b krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527125, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 16:38:45 dist-test-slave-z40b krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527125, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 16:38:45.933521 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:45.926387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51066 (local address 127.0.98.1:33839)
0927 16:38:45.926501 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:45.926504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:45.926645 (+ 141us) server_negotiation.cc:407] Connection header received
0927 16:38:45.926717 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:45.926719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:45.926751 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:45.926811 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:45.927418 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.927880 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.928634 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.928823 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.928892 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:45.931000 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:45.931003 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:45.931005 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:45.931024 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:45.932197 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:45.932527 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:45.932529 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:45.932530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:45.932589 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:45.932783 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:45.932785 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:45.932785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:45.932946 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:45.933036 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:45.933235 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 16:38:45.933317 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 16:38:45 dist-test-slave-z40b krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 16:38:45 dist-test-slave-z40b krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 16:38:45.955477 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:45.948031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51074 (local address 127.0.98.1:33839)
0927 16:38:45.948165 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:45.948170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:45.948309 (+ 139us) server_negotiation.cc:407] Connection header received
0927 16:38:45.948461 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:45.948465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:45.948503 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:45.948582 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:45.949189 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.949649 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.950457 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:45.950655 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:45.950767 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:45.952770 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:45.952773 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:45.952777 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:45.952801 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:45.953854 (+ 1053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:45.954299 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:45.954304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:45.954305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:45.954380 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:45.954669 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:45.954673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:45.954675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:45.954817 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:45.954940 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:45.955162 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 16:38:45.955305 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 16:38:45.956741 1067 tablet_service.cc:2908] Rejecting scan request for tablet 1332d5ebe43e467bbe611eb92e0be5dd: Uninitialized: clean time has not yet been initialized
I20260927 16:38:45.958770 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 1024
I20260927 16:38:45.965391 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 1158
I20260927 16:38:45.970825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 1292
I20260927 16:38:45.975740 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 930
2026-09-27T16:38:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2412 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:38:46 dist-test-slave-z40b krb5kdc[1463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:38:46 dist-test-slave-z40b krb5kdc[1463](info): set up 2 sockets
Sep 27 16:38:46 dist-test-slave-z40b krb5kdc[1463](info): commencing operation
krb5kdc: starting...
W20260927 16:38:46.987043 392 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:38:47 dist-test-slave-z40b krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:38:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:38:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:38:47.098610 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36147
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:38:47.199481 1476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:38:47.199734 1476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:38:47.199770 1476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:38:47.204133 1476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:38:47.204180 1476 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 16:38:47.204195 1476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:38:47.204228 1476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:38:47.204243 1476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:38:47.204258 1476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:38:47.209194 1476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36147
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:38:47.210060 1476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:38:47.210915 1476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:38:47.218766 1482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:47.218776 1484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:47.218791 1481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:47.218871 1476 server_base.cc:1034] running on GCE node
I20260927 16:38:47.219615 1476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:38:47.220562 1476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:38:47.221712 1476 hybrid_clock.cc:648] HybridClock initialized: now 1790527127221686 us; error 39 us; skew 500 ppm
Sep 27 16:38:47 dist-test-slave-z40b krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:38:47.224357 1476 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:38:47.225226 1476 webserver.cc:460] Webserver started at http://127.0.98.62:39341/ using document root <none> and password file <none>
I20260927 16:38:47.225749 1476 fs_manager.cc:359] Metadata directory not provided
I20260927 16:38:47.225807 1476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:38:47.226040 1476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:38:47.228078 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "fb458acfde4b459aa26e0b6170f5c188"
format_stamp: "Formatted at 2026-09-27 16:38:47 on dist-test-slave-z40b"
server_key: "6bdf6c830b15641344eb90af22374c9a"
server_key_iv: "8740f35a7f52f24130cc78d1893f958f"
server_key_version: "encryptionkey@0"
I20260927 16:38:47.228643 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "fb458acfde4b459aa26e0b6170f5c188"
format_stamp: "Formatted at 2026-09-27 16:38:47 on dist-test-slave-z40b"
server_key: "6bdf6c830b15641344eb90af22374c9a"
server_key_iv: "8740f35a7f52f24130cc78d1893f958f"
server_key_version: "encryptionkey@0"
I20260927 16:38:47.232191 1476 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:38:47.234179 1491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:47.235294 1476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:38:47.235425 1476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "fb458acfde4b459aa26e0b6170f5c188"
format_stamp: "Formatted at 2026-09-27 16:38:47 on dist-test-slave-z40b"
server_key: "6bdf6c830b15641344eb90af22374c9a"
server_key_iv: "8740f35a7f52f24130cc78d1893f958f"
server_key_version: "encryptionkey@0"
I20260927 16:38:47.235549 1476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:38:47.253475 1476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:38:47.256116 1476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:38:47.256299 1476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:38:47.264029 1476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36147
I20260927 16:38:47.264057 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36147 every 8 connection(s)
I20260927 16:38:47.264962 1476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:38:47.267575 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:47.272778 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb458acfde4b459aa26e0b6170f5c188: Bootstrap starting.
I20260927 16:38:47.273799 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 1476
I20260927 16:38:47.273931 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:38:47.274184 392 external_mini_cluster.cc:1428] Setting key 41f546a9213f4e396ec1ba85081d66b0
I20260927 16:38:47.275509 1554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb458acfde4b459aa26e0b6170f5c188: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:47.276192 1554 log.cc:824] T 00000000000000000000000000000000 P fb458acfde4b459aa26e0b6170f5c188: Log is configured to *not* fsync() on all Append() calls
I20260927 16:38:47.277947 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb458acfde4b459aa26e0b6170f5c188: No bootstrap required, opened a new log
I20260927 16:38:47.281195 1554 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb458acfde4b459aa26e0b6170f5c188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb458acfde4b459aa26e0b6170f5c188" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36147 } }
I20260927 16:38:47.281381 1554 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb458acfde4b459aa26e0b6170f5c188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:47.281414 1554 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb458acfde4b459aa26e0b6170f5c188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb458acfde4b459aa26e0b6170f5c188, State: Initialized, Role: FOLLOWER
I20260927 16:38:47.281816 1554 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb458acfde4b459aa26e0b6170f5c188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb458acfde4b459aa26e0b6170f5c188" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36147 } }
I20260927 16:38:47.281924 1554 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb458acfde4b459aa26e0b6170f5c188 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:38:47.282024 1554 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb458acfde4b459aa26e0b6170f5c188 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:38:47.282150 1554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb458acfde4b459aa26e0b6170f5c188 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 16:38:47 dist-test-slave-z40b krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:38:47.282981 1554 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb458acfde4b459aa26e0b6170f5c188 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb458acfde4b459aa26e0b6170f5c188" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36147 } }
I20260927 16:38:47.283324 1554 leader_election.cc:302] T 00000000000000000000000000000000 P fb458acfde4b459aa26e0b6170f5c188 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb458acfde4b459aa26e0b6170f5c188; no voters:
I20260927 16:38:47.283650 1554 leader_election.cc:288] T 00000000000000000000000000000000 P fb458acfde4b459aa26e0b6170f5c188 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:38:47.283874 1559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb458acfde4b459aa26e0b6170f5c188 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:38:47.284144 1559 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb458acfde4b459aa26e0b6170f5c188 [term 1 LEADER]: Becoming Leader. State: Replica: fb458acfde4b459aa26e0b6170f5c188, State: Running, Role: LEADER
I20260927 16:38:47.284476 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb458acfde4b459aa26e0b6170f5c188 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb458acfde4b459aa26e0b6170f5c188" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36147 } }
I20260927 16:38:47.284937 1554 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb458acfde4b459aa26e0b6170f5c188 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:38:47.287746 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb458acfde4b459aa26e0b6170f5c188 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb458acfde4b459aa26e0b6170f5c188. Latest consensus state: current_term: 1 leader_uuid: "fb458acfde4b459aa26e0b6170f5c188" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb458acfde4b459aa26e0b6170f5c188" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36147 } } }
I20260927 16:38:47.287801 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb458acfde4b459aa26e0b6170f5c188 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb458acfde4b459aa26e0b6170f5c188" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb458acfde4b459aa26e0b6170f5c188" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36147 } } }
I20260927 16:38:47.287868 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb458acfde4b459aa26e0b6170f5c188 [sys.catalog]: This master's current role is: LEADER
I20260927 16:38:47.287906 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb458acfde4b459aa26e0b6170f5c188 [sys.catalog]: This master's current role is: LEADER
I20260927 16:38:47.288558 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:47.275289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36426 (local address 127.0.98.62:36147)
0927 16:38:47.275730 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:47.275744 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:47.275777 (+ 33us) server_negotiation.cc:407] Connection header received
0927 16:38:47.276416 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:47.276433 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:47.276823 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:47.277168 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:47.277927 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:47.278950 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:47.279883 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:47.280357 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:47.280529 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:47.283119 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:47.283125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:47.283139 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:47.283173 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:47.285213 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:47.285709 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:47.285715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:47.285721 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:47.285848 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:47.286200 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:47.286203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:47.286204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:47.286652 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:47.286848 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:47.287150 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 16:38:47.287506 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":126,"threads_started":1}
W20260927 16:38:47.289944 1574 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb458acfde4b459aa26e0b6170f5c188: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:38:47.290012 1574 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:38:47.290083 1575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:38:47.290840 1575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:38:47.295680 1575 catalog_manager.cc:1347] Generated new cluster ID: 8b9f16f6a745484c8a7b528fdd4c85d7
I20260927 16:38:47.295735 1575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:38:47.318738 1575 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:38:47.320001 1575 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:38:47.327991 1575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb458acfde4b459aa26e0b6170f5c188: Generated new TSK 0
I20260927 16:38:47.328673 1575 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:38:47.354025 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36147
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:38:47.489953 1580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:38:47.490177 1580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:38:47.490252 1580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:38:47.494005 1580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:38:47.494051 1580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:38:47.494086 1580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 16:38:47.494113 1580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:38:47.499074 1580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:38:47.500000 1580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:38:47.500912 1580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:38:47.509626 1585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:47.509722 1580 server_base.cc:1034] running on GCE node
W20260927 16:38:47.509590 1588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:47.509616 1586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:47.510218 1580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:38:47.510740 1580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:38:47.511886 1580 hybrid_clock.cc:648] HybridClock initialized: now 1790527127511856 us; error 42 us; skew 500 ppm
Sep 27 16:38:47 dist-test-slave-z40b krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:38:47.514477 1580 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:38:47.515327 1580 webserver.cc:460] Webserver started at http://127.0.98.1:33673/ using document root <none> and password file <none>
I20260927 16:38:47.515820 1580 fs_manager.cc:359] Metadata directory not provided
I20260927 16:38:47.515875 1580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:38:47.516031 1580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:38:47.517841 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-0/data/instance:
uuid: "1e58dbf1f0174d44955ed3d20569f53f"
format_stamp: "Formatted at 2026-09-27 16:38:47 on dist-test-slave-z40b"
server_key: "72500eaed171d14f97bbf0ceac65a4ea"
server_key_iv: "f292d7d1e60368a0d394395296fb225c"
server_key_version: "encryptionkey@0"
I20260927 16:38:47.518338 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1e58dbf1f0174d44955ed3d20569f53f"
format_stamp: "Formatted at 2026-09-27 16:38:47 on dist-test-slave-z40b"
server_key: "72500eaed171d14f97bbf0ceac65a4ea"
server_key_iv: "f292d7d1e60368a0d394395296fb225c"
server_key_version: "encryptionkey@0"
I20260927 16:38:47.521721 1580 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 16:38:47.523823 1595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:47.524919 1580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:38:47.525040 1580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "1e58dbf1f0174d44955ed3d20569f53f"
format_stamp: "Formatted at 2026-09-27 16:38:47 on dist-test-slave-z40b"
server_key: "72500eaed171d14f97bbf0ceac65a4ea"
server_key_iv: "f292d7d1e60368a0d394395296fb225c"
server_key_version: "encryptionkey@0"
I20260927 16:38:47.525133 1580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:38:47.541354 1580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:38:47.544421 1580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:38:47.544595 1580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:38:47.545994 1580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:38:47.546085 1580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:47.546141 1580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:38:47.546169 1580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 16:38:47 dist-test-slave-z40b krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:38:47.556568 1580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42325
I20260927 16:38:47.557180 1710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42325 every 8 connection(s)
I20260927 16:38:47.557456 1580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:38:47.559036 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:47.546542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57649 (local address 127.0.98.62:36147)
0927 16:38:47.546647 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:47.546651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:47.547610 (+ 959us) server_negotiation.cc:407] Connection header received
0927 16:38:47.548495 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:47.548498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:47.548555 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:47.548660 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:47.549994 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:47.550738 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:47.552093 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:47.552324 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:47.552429 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:47.555680 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:47.555683 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:47.555686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:47.555710 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:47.557037 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:47.557619 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:47.557622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:47.557624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:47.557708 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:47.558100 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:47.558103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:47.558104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:47.558307 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:47.558397 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:47.558788 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 16:38:47.558896 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 16:38:47.559744 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 1580
I20260927 16:38:47.559752 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:47.547020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36147 (local address 127.0.98.1:57649)
0927 16:38:47.547477 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:47.547514 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:47.548304 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:47.548819 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:47.548828 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:47.549216 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:47.549818 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:47.549836 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:47.550850 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:47.550854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:47.551977 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:47.551986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:47.552461 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:47.552465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:47.552671 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:47.553200 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:47.553224 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:47.555511 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:47.557139 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:47.557147 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:47.557158 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:47.557486 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:47.557815 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:47.557819 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:47.557821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:47.557984 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:47.558393 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:47.558397 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:47.558589 (+ 192us) client_negotiation.cc:769] Sending connection context
0927 16:38:47.558729 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 16:38:47.558951 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":145,"threads_started":1}
I20260927 16:38:47.559826 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:38:47.560070 392 external_mini_cluster.cc:1428] Setting key 587a2484fb5bfb65bd91dae4864f8ec0
I20260927 16:38:47.561460 1711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36147
I20260927 16:38:47.561589 1711 heartbeater.cc:464] Registering TS with master...
I20260927 16:38:47.562134 1711 heartbeater.cc:511] Master 127.0.98.62:36147 requested a full tablet report, sending...
I20260927 16:38:47.564708 1508 ts_manager.cc:194] Registered new tserver with Master: 1e58dbf1f0174d44955ed3d20569f53f (127.0.98.1:42325)
I20260927 16:38:47.566200 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57649
I20260927 16:38:47.571264 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:47.563538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56089 (local address 127.0.98.62:36147)
0927 16:38:47.563637 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:47.563642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:47.563833 (+ 191us) server_negotiation.cc:407] Connection header received
0927 16:38:47.563961 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:47.563975 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:47.564012 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:47.564072 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:47.564544 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:47.565120 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:47.566233 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:47.566589 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:47.566718 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:47.567874 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:47.567881 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:47.567885 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:47.567917 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:47.569640 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:47.570079 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:47.570084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:47.570085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:47.570161 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:47.570446 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:47.570449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:47.570450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:47.570604 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:47.570704 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:47.570986 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 16:38:47.571095 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 16:38:47.571274 1718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:47.563468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36147 (local address 127.0.98.1:56089)
0927 16:38:47.563751 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:47.563762 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:47.563885 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:47.564163 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:47.564165 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:47.564341 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:47.564439 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:47.564446 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:47.565244 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:47.565247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:47.566126 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:47.566135 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:47.566723 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:47.566727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:47.566828 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:47.567302 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:47.567332 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:47.567731 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:47.569759 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:47.569762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:47.569763 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:47.569982 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:47.570258 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:47.570261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:47.570263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:47.570343 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:47.570706 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:47.570710 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:47.570798 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 16:38:47.570917 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 16:38:47.571070 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:38:47.590759 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36147
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:38:47.693881 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:38:47.694113 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:38:47.694150 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:38:47.698146 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:38:47.698194 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:38:47.698261 1720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 16:38:47.698344 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:38:47.703171 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:38:47.704131 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:38:47.705147 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:38:47.714000 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:47.714028 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:47.714141 1720 server_base.cc:1034] running on GCE node
W20260927 16:38:47.714016 1725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:47.714618 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:38:47.715243 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:38:47.716409 1720 hybrid_clock.cc:648] HybridClock initialized: now 1790527127716391 us; error 29 us; skew 500 ppm
Sep 27 16:38:47 dist-test-slave-z40b krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:38:47.718990 1720 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:38:47.719867 1720 webserver.cc:460] Webserver started at http://127.0.98.2:37935/ using document root <none> and password file <none>
I20260927 16:38:47.720388 1720 fs_manager.cc:359] Metadata directory not provided
I20260927 16:38:47.720453 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:38:47.720724 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:38:47.722609 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-1/data/instance:
uuid: "5fa951dba6ac49e398a13449b31694c0"
format_stamp: "Formatted at 2026-09-27 16:38:47 on dist-test-slave-z40b"
server_key: "a2f2b77694343265f720dcda821326af"
server_key_iv: "79ae2c746adff2b35a68218d4642e4d4"
server_key_version: "encryptionkey@0"
I20260927 16:38:47.723135 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5fa951dba6ac49e398a13449b31694c0"
format_stamp: "Formatted at 2026-09-27 16:38:47 on dist-test-slave-z40b"
server_key: "a2f2b77694343265f720dcda821326af"
server_key_iv: "79ae2c746adff2b35a68218d4642e4d4"
server_key_version: "encryptionkey@0"
I20260927 16:38:47.726502 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 16:38:47.728855 1735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:47.729987 1720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 16:38:47.730113 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "5fa951dba6ac49e398a13449b31694c0"
format_stamp: "Formatted at 2026-09-27 16:38:47 on dist-test-slave-z40b"
server_key: "a2f2b77694343265f720dcda821326af"
server_key_iv: "79ae2c746adff2b35a68218d4642e4d4"
server_key_version: "encryptionkey@0"
I20260927 16:38:47.730209 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:38:47.746850 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:38:47.749362 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:38:47.749542 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:38:47.750780 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:38:47.750849 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:47.750908 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:38:47.750962 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 16:38:47 dist-test-slave-z40b krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:38:47.761513 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34885
I20260927 16:38:47.761873 1850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34885 every 8 connection(s)
I20260927 16:38:47.762624 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:38:47.764565 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:47.751568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33437 (local address 127.0.98.62:36147)
0927 16:38:47.751681 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:47.751685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:47.752607 (+ 922us) server_negotiation.cc:407] Connection header received
0927 16:38:47.753473 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:47.753477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:47.753531 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:47.753608 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:47.754672 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:47.755421 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:47.756751 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:47.756988 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:47.757100 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:47.760901 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:47.760904 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:47.760906 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:47.760928 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:47.762573 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:47.763122 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:47.763125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:47.763127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:47.763256 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:47.763602 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:47.763605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:47.763607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:47.763797 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:47.763915 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:47.764344 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 16:38:47.764452 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 16:38:47.765234 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:47.752050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36147 (local address 127.0.98.2:33437)
0927 16:38:47.752470 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:47.752516 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:47.753285 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:47.753761 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:47.753769 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:47.754125 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:47.754532 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:47.754544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:47.755541 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:47.755550 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:47.756626 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:47.756635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:47.757133 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:47.757137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:47.757331 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:47.757858 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:47.757883 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:47.760680 (+ 2797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:47.762662 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:47.762668 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:47.762680 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:47.763004 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:47.763388 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:47.763391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:47.763393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:47.763524 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:47.763924 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:47.763928 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:47.764124 (+ 196us) client_negotiation.cc:769] Sending connection context
0927 16:38:47.764264 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 16:38:47.764473 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":119,"threads_started":1}
I20260927 16:38:47.765676 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 1720
I20260927 16:38:47.765792 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:38:47.766134 392 external_mini_cluster.cc:1428] Setting key 88d89d5cbe1e184fdd0af6f0a8390c85
I20260927 16:38:47.766574 1851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36147
I20260927 16:38:47.766707 1851 heartbeater.cc:464] Registering TS with master...
I20260927 16:38:47.767273 1851 heartbeater.cc:511] Master 127.0.98.62:36147 requested a full tablet report, sending...
I20260927 16:38:47.768373 1506 ts_manager.cc:194] Registered new tserver with Master: 5fa951dba6ac49e398a13449b31694c0 (127.0.98.2:34885)
I20260927 16:38:47.769096 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33437
I20260927 16:38:47.775164 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:47.768788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42373 (local address 127.0.98.62:36147)
0927 16:38:47.768871 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:47.768874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:47.768885 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:38:47.768968 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:47.768970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:47.769002 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:47.769072 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:47.769543 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:47.769993 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:47.770743 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:47.770994 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:47.771093 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:47.772149 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:47.772153 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:47.772155 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:47.772177 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:47.773549 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:47.773970 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:47.773973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:47.773976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:47.774056 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:47.774330 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:47.774334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:47.774335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:47.774527 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:47.774629 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:47.774907 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 16:38:47.775017 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 16:38:47.775194 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:47.768509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36147 (local address 127.0.98.2:42373)
0927 16:38:47.768785 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:47.768802 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:47.768893 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:47.769109 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:47.769114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:47.769319 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:47.769435 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:47.769441 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:47.770108 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:47.770111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:47.770641 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:47.770647 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:47.771118 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:47.771121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:47.771242 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:47.771707 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:47.771720 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:47.772014 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:47.773678 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:47.773681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:47.773684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:47.773867 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:47.774146 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:47.774150 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:47.774152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:47.774236 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:47.774631 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:47.774634 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:47.774719 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 16:38:47.774849 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 16:38:47.775023 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:38:47.791661 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36147
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:38:47.907791 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:38:47.908015 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:38:47.908082 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:38:47.911782 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:38:47.911825 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:38:47.911860 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 16:38:47.911939 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:38:47.916498 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:38:47.917382 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:38:47.918294 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:38:47.926978 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:47.926968 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:47.927143 1860 server_base.cc:1034] running on GCE node
W20260927 16:38:47.926968 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:47.927757 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:38:47.928328 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:38:47.929481 1860 hybrid_clock.cc:648] HybridClock initialized: now 1790527127929467 us; error 27 us; skew 500 ppm
Sep 27 16:38:47 dist-test-slave-z40b krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:38:47.932045 1860 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:38:47.932842 1860 webserver.cc:460] Webserver started at http://127.0.98.3:46611/ using document root <none> and password file <none>
I20260927 16:38:47.933353 1860 fs_manager.cc:359] Metadata directory not provided
I20260927 16:38:47.933468 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:38:47.933748 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:38:47.935638 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-2/data/instance:
uuid: "913c607885ed48c78f673a746ec084d8"
format_stamp: "Formatted at 2026-09-27 16:38:47 on dist-test-slave-z40b"
server_key: "613c332c83dbc939c9c660d27f95aeff"
server_key_iv: "4ac84d8f8ed439065e90b725d783f2b4"
server_key_version: "encryptionkey@0"
I20260927 16:38:47.936203 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-2/wal/instance:
uuid: "913c607885ed48c78f673a746ec084d8"
format_stamp: "Formatted at 2026-09-27 16:38:47 on dist-test-slave-z40b"
server_key: "613c332c83dbc939c9c660d27f95aeff"
server_key_iv: "4ac84d8f8ed439065e90b725d783f2b4"
server_key_version: "encryptionkey@0"
I20260927 16:38:47.939702 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:38:47.941829 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:47.942869 1860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:38:47.942998 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "913c607885ed48c78f673a746ec084d8"
format_stamp: "Formatted at 2026-09-27 16:38:47 on dist-test-slave-z40b"
server_key: "613c332c83dbc939c9c660d27f95aeff"
server_key_iv: "4ac84d8f8ed439065e90b725d783f2b4"
server_key_version: "encryptionkey@0"
I20260927 16:38:47.943096 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:38:47.959023 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:38:47.961524 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:38:47.961691 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:38:47.962893 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:38:47.962960 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:47.963030 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:38:47.963078 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 16:38:47 dist-test-slave-z40b krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:38:47.973795 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43199
I20260927 16:38:47.974099 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43199 every 8 connection(s)
I20260927 16:38:47.974588 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:38:47.976431 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:47.963675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58481 (local address 127.0.98.62:36147)
0927 16:38:47.963790 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:47.963793 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:47.964641 (+ 848us) server_negotiation.cc:407] Connection header received
0927 16:38:47.965520 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:47.965524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:47.965581 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:47.965684 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:47.967061 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:47.967948 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:47.969631 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:47.969996 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:47.970127 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:47.973017 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:47.973021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:47.973023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:47.973049 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:47.974530 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:47.975018 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:47.975021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:47.975022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:47.975097 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:47.975457 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:47.975461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:47.975463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:47.975635 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:47.975742 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:47.976175 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 16:38:47.976280 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 16:38:47.976781 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 1860
I20260927 16:38:47.976886 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:38:47.977144 392 external_mini_cluster.cc:1428] Setting key 4b161906a9f1e313e3ec4af855bf84d5
I20260927 16:38:47.977243 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:47.964100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36147 (local address 127.0.98.3:58481)
0927 16:38:47.964516 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:47.964550 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:47.965311 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:47.965862 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:47.965872 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:47.966283 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:47.966844 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:47.966859 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:47.968065 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:47.968071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:47.969501 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:47.969511 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:47.970189 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:47.970194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:47.970387 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:47.970971 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:47.970988 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:47.972870 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:47.974636 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:47.974641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:47.974653 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:47.974888 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:47.975230 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:47.975233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:47.975235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:47.975376 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:47.975751 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:47.975758 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:47.975966 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 16:38:47.976138 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 16:38:47.976399 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":122,"threads_started":1}
I20260927 16:38:47.978411 1991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36147
I20260927 16:38:47.978567 1991 heartbeater.cc:464] Registering TS with master...
I20260927 16:38:47.979091 1991 heartbeater.cc:511] Master 127.0.98.62:36147 requested a full tablet report, sending...
I20260927 16:38:47.980137 1506 ts_manager.cc:194] Registered new tserver with Master: 913c607885ed48c78f673a746ec084d8 (127.0.98.3:43199)
I20260927 16:38:47.980787 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58481
I20260927 16:38:47.986439 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:47.980534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57579 (local address 127.0.98.62:36147)
0927 16:38:47.980618 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:47.980623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:47.980772 (+ 149us) server_negotiation.cc:407] Connection header received
0927 16:38:47.980870 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:47.980873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:47.980915 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:47.980989 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:47.981387 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:47.981833 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:47.982561 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:47.982753 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:47.982857 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:47.983741 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:47.983744 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:47.983747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:47.983773 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:47.984910 (+ 1137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:47.985298 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:47.985302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:47.985304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:47.985376 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:47.985654 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:47.985659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:47.985661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:47.985825 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:47.985923 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:47.986199 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 16:38:47.986310 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 16:38:47.986436 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:47.980485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36147 (local address 127.0.98.3:57579)
0927 16:38:47.980693 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:47.980711 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:47.980805 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:47.981008 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:47.981011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:47.981198 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:47.981288 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:47.981293 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:47.981949 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:47.981952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:47.982454 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:47.982461 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:47.982858 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:47.982861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:47.982935 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:47.983369 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:47.983378 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:47.983623 (+ 245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:47.985019 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:47.985022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:47.985024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:47.985204 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:47.985464 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:47.985469 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:47.985470 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:47.985568 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:47.985924 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:47.985928 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:47.986023 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 16:38:47.986127 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 16:38:47.986266 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":75,"threads_started":1}
I20260927 16:38:47.990696 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 16:38:47 dist-test-slave-z40b krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527127, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 16:38:48 dist-test-slave-z40b krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527127, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:38:48.014319 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.006911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36430 (local address 127.0.98.62:36147)
0927 16:38:48.007028 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:48.007033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:48.007046 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:38:48.007081 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:48.007083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:48.007118 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:48.007185 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:48.007857 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.008387 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.009238 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.009446 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.009534 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.011767 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:48.011770 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:48.011772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:48.011793 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:48.012965 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:48.013304 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:48.013307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:48.013308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:48.013382 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:48.013613 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:48.013616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:48.013617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:48.013760 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:48.013852 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:48.014020 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 16:38:48.014098 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:38:48.016897 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:38:48.019727 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:38:48.033650 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.028459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44104 (local address 127.0.98.2:34885)
0927 16:38:48.029761 (+ 1302us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:48.029772 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:48.029788 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:38:48.029842 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:48.029845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:48.029983 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:48.030120 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:38:48.030314 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.031165 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.032278 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.033120 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.033230 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.033380 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:48.033436 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 16:38:48.033514 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1218,"thread_start_us":112,"threads_started":1}
I20260927 16:38:48.033660 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.028275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34885 (local address 127.0.0.1:44104)
0927 16:38:48.029458 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:48.029471 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:48.029539 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:48.030102 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:48.030104 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:48.030120 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:38:48.030205 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:48.030212 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.031297 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.031303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:48.032149 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:48.032158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.033225 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.033228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:48.033298 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.033325 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 16:38:48.033396 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 16:38:48.033510 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1128,"mutex_wait_us":50,"thread_start_us":85,"threads_started":1}
I20260927 16:38:48.034206 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.027594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42325 (local address 127.0.0.1:49052)
0927 16:38:48.028519 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:48.028565 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:48.028696 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:48.029258 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:48.029266 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:48.029287 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:38:48.029424 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:48.029431 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.030812 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.030815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:48.032284 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:48.032294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.033798 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.033801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:48.033883 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.033951 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 16:38:48.034022 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 16:38:48.034084 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":39,"spinlock_wait_cycles":4608,"thread_start_us":61,"threads_started":1}
I20260927 16:38:48.034643 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.027771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49052 (local address 127.0.98.1:42325)
0927 16:38:48.028767 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:48.028782 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:48.028809 (+ 27us) server_negotiation.cc:407] Connection header received
0927 16:38:48.028889 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:48.028892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:48.029050 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:48.029258 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:38:48.029554 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.030689 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.032410 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.033120 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.033230 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.033386 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:48.034390 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0927 16:38:48.034478 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":133,"threads_started":1}
I20260927 16:38:48.034775 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.027950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43199 (local address 127.0.0.1:39524)
0927 16:38:48.028521 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:48.028567 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:48.028695 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:48.029425 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:48.029428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:48.029444 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:38:48.029540 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:48.029546 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.030812 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.030815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:48.032291 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:48.032296 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.034386 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.034389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:48.034485 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.034503 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 16:38:48.034571 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 16:38:48.034641 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":119,"spinlock_wait_cycles":4096,"thread_start_us":87,"threads_started":1}
I20260927 16:38:48.035380 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.028598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39524 (local address 127.0.98.3:43199)
0927 16:38:48.028984 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:48.028992 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:48.029009 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:38:48.029065 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:48.029068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:48.029281 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:48.029426 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:38:48.029629 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.030691 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.033465 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.034269 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.034377 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.034504 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:48.034592 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 16:38:48.034705 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":106,"threads_started":1}
I20260927 16:38:48.036876 1645 tablet_service.cc:1467] Processing CreateTablet for tablet ad903d9025f741fb80eb4c4b5820fd2e (DEFAULT_TABLE table=test-table [id=1935c23bf97a4f92a881df0a34ba6d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:38:48.036819 1785 tablet_service.cc:1467] Processing CreateTablet for tablet ad903d9025f741fb80eb4c4b5820fd2e (DEFAULT_TABLE table=test-table [id=1935c23bf97a4f92a881df0a34ba6d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:38:48.037243 1925 tablet_service.cc:1467] Processing CreateTablet for tablet ad903d9025f741fb80eb4c4b5820fd2e (DEFAULT_TABLE table=test-table [id=1935c23bf97a4f92a881df0a34ba6d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:38:48.037896 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad903d9025f741fb80eb4c4b5820fd2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:48.037896 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad903d9025f741fb80eb4c4b5820fd2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:48.038208 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad903d9025f741fb80eb4c4b5820fd2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:48.044989 2014 tablet_bootstrap.cc:492] T ad903d9025f741fb80eb4c4b5820fd2e P 913c607885ed48c78f673a746ec084d8: Bootstrap starting.
I20260927 16:38:48.045377 2015 tablet_bootstrap.cc:492] T ad903d9025f741fb80eb4c4b5820fd2e P 1e58dbf1f0174d44955ed3d20569f53f: Bootstrap starting.
I20260927 16:38:48.046271 2016 tablet_bootstrap.cc:492] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0: Bootstrap starting.
I20260927 16:38:48.047399 2015 tablet_bootstrap.cc:654] T ad903d9025f741fb80eb4c4b5820fd2e P 1e58dbf1f0174d44955ed3d20569f53f: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:48.047397 2014 tablet_bootstrap.cc:654] T ad903d9025f741fb80eb4c4b5820fd2e P 913c607885ed48c78f673a746ec084d8: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:48.048254 2016 tablet_bootstrap.cc:654] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:48.048274 2014 log.cc:824] T ad903d9025f741fb80eb4c4b5820fd2e P 913c607885ed48c78f673a746ec084d8: Log is configured to *not* fsync() on all Append() calls
I20260927 16:38:48.048274 2015 log.cc:824] T ad903d9025f741fb80eb4c4b5820fd2e P 1e58dbf1f0174d44955ed3d20569f53f: Log is configured to *not* fsync() on all Append() calls
I20260927 16:38:48.049139 2016 log.cc:824] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0: Log is configured to *not* fsync() on all Append() calls
I20260927 16:38:48.050307 2015 tablet_bootstrap.cc:492] T ad903d9025f741fb80eb4c4b5820fd2e P 1e58dbf1f0174d44955ed3d20569f53f: No bootstrap required, opened a new log
I20260927 16:38:48.050307 2014 tablet_bootstrap.cc:492] T ad903d9025f741fb80eb4c4b5820fd2e P 913c607885ed48c78f673a746ec084d8: No bootstrap required, opened a new log
I20260927 16:38:48.050484 2014 ts_tablet_manager.cc:1397] T ad903d9025f741fb80eb4c4b5820fd2e P 913c607885ed48c78f673a746ec084d8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260927 16:38:48.050486 2015 ts_tablet_manager.cc:1397] T ad903d9025f741fb80eb4c4b5820fd2e P 1e58dbf1f0174d44955ed3d20569f53f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260927 16:38:48.050884 2016 tablet_bootstrap.cc:492] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0: No bootstrap required, opened a new log
I20260927 16:38:48.051069 2016 ts_tablet_manager.cc:1397] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 16:38:48.053668 2016 raft_consensus.cc:348] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.053900 2016 raft_consensus.cc:374] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:48.053972 2016 raft_consensus.cc:729] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa951dba6ac49e398a13449b31694c0, State: Initialized, Role: FOLLOWER
I20260927 16:38:48.054208 2015 raft_consensus.cc:348] T ad903d9025f741fb80eb4c4b5820fd2e P 1e58dbf1f0174d44955ed3d20569f53f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.054208 2014 raft_consensus.cc:348] T ad903d9025f741fb80eb4c4b5820fd2e P 913c607885ed48c78f673a746ec084d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.054430 2014 raft_consensus.cc:374] T ad903d9025f741fb80eb4c4b5820fd2e P 913c607885ed48c78f673a746ec084d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:48.054505 2014 raft_consensus.cc:729] T ad903d9025f741fb80eb4c4b5820fd2e P 913c607885ed48c78f673a746ec084d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913c607885ed48c78f673a746ec084d8, State: Initialized, Role: FOLLOWER
I20260927 16:38:48.054435 2016 consensus_queue.cc:260] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.054644 2015 raft_consensus.cc:374] T ad903d9025f741fb80eb4c4b5820fd2e P 1e58dbf1f0174d44955ed3d20569f53f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:48.054730 2015 raft_consensus.cc:729] T ad903d9025f741fb80eb4c4b5820fd2e P 1e58dbf1f0174d44955ed3d20569f53f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e58dbf1f0174d44955ed3d20569f53f, State: Initialized, Role: FOLLOWER
I20260927 16:38:48.054908 2014 consensus_queue.cc:260] T ad903d9025f741fb80eb4c4b5820fd2e P 913c607885ed48c78f673a746ec084d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.055217 2015 consensus_queue.cc:260] T ad903d9025f741fb80eb4c4b5820fd2e P 1e58dbf1f0174d44955ed3d20569f53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.055431 2016 ts_tablet_manager.cc:1428] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 16:38:48.055601 1991 heartbeater.cc:503] Master 127.0.98.62:36147 was elected leader, sending a full tablet report...
I20260927 16:38:48.055757 1851 heartbeater.cc:503] Master 127.0.98.62:36147 was elected leader, sending a full tablet report...
I20260927 16:38:48.055850 2014 ts_tablet_manager.cc:1428] T ad903d9025f741fb80eb4c4b5820fd2e P 913c607885ed48c78f673a746ec084d8: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260927 16:38:48.056030 2015 ts_tablet_manager.cc:1428] T ad903d9025f741fb80eb4c4b5820fd2e P 1e58dbf1f0174d44955ed3d20569f53f: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260927 16:38:48.056330 1711 heartbeater.cc:503] Master 127.0.98.62:36147 was elected leader, sending a full tablet report...
W20260927 16:38:48.058667 1712 tablet.cc:2369] T ad903d9025f741fb80eb4c4b5820fd2e P 1e58dbf1f0174d44955ed3d20569f53f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:38:48.113770 2020 raft_consensus.cc:482] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:38:48.113937 2020 raft_consensus.cc:504] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.115056 2020 leader_election.cc:288] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e58dbf1f0174d44955ed3d20569f53f (127.0.98.1:42325), 913c607885ed48c78f673a746ec084d8 (127.0.98.3:43199)
I20260927 16:38:48.118991 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.115354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54601 (local address 127.0.98.1:42325)
0927 16:38:48.115478 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:48.115482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:48.115510 (+ 28us) server_negotiation.cc:407] Connection header received
0927 16:38:48.115591 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:48.115594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:48.115643 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:48.115726 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:38:48.115979 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.116478 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.118002 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.118396 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.118522 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.118589 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:48.118810 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 16:38:48.118880 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 16:38:48.118993 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.115422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36763 (local address 127.0.98.3:43199)
0927 16:38:48.115563 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:48.115568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:48.115976 (+ 408us) server_negotiation.cc:407] Connection header received
0927 16:38:48.116120 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:48.116123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:48.116172 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:48.116250 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:38:48.116506 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.117028 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.118008 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.118386 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.118498 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.118564 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:48.118802 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 16:38:48.118870 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 16:38:48.119015 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.115269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42325 (local address 127.0.98.2:54601)
0927 16:38:48.115391 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:48.115409 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:48.115497 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:48.115741 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:48.115743 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:48.115761 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:38:48.115873 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:48.115878 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.116596 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.116601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:48.117874 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:48.117883 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.118514 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.118518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:48.118687 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.118711 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 16:38:48.118795 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 16:38:48.118871 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":28}
I20260927 16:38:48.119017 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.115586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43199 (local address 127.0.98.2:36763)
0927 16:38:48.115878 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:48.115889 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:48.116016 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:48.116258 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:48.116261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:48.116275 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:38:48.116390 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:48.116394 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.117130 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.117137 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:48.117884 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:48.117892 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.118501 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.118504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:48.118677 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.118703 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 16:38:48.118787 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 16:38:48.118853 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":38,"thread_start_us":104,"threads_started":1}
I20260927 16:38:48.119731 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad903d9025f741fb80eb4c4b5820fd2e" candidate_uuid: "5fa951dba6ac49e398a13449b31694c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e58dbf1f0174d44955ed3d20569f53f" is_pre_election: true
I20260927 16:38:48.119742 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad903d9025f741fb80eb4c4b5820fd2e" candidate_uuid: "5fa951dba6ac49e398a13449b31694c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913c607885ed48c78f673a746ec084d8" is_pre_election: true
I20260927 16:38:48.120019 1945 raft_consensus.cc:2454] T ad903d9025f741fb80eb4c4b5820fd2e P 913c607885ed48c78f673a746ec084d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fa951dba6ac49e398a13449b31694c0 in term 0.
I20260927 16:38:48.120019 1665 raft_consensus.cc:2454] T ad903d9025f741fb80eb4c4b5820fd2e P 1e58dbf1f0174d44955ed3d20569f53f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fa951dba6ac49e398a13449b31694c0 in term 0.
I20260927 16:38:48.120532 1737 leader_election.cc:302] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fa951dba6ac49e398a13449b31694c0, 913c607885ed48c78f673a746ec084d8; no voters:
I20260927 16:38:48.120836 2020 raft_consensus.cc:2781] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:38:48.120956 2020 raft_consensus.cc:482] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:38:48.121018 2020 raft_consensus.cc:3037] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:48.122056 2020 raft_consensus.cc:504] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.122375 2020 leader_election.cc:288] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0 [CANDIDATE]: Term 1 election: Requested vote from peers 1e58dbf1f0174d44955ed3d20569f53f (127.0.98.1:42325), 913c607885ed48c78f673a746ec084d8 (127.0.98.3:43199)
I20260927 16:38:48.122715 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad903d9025f741fb80eb4c4b5820fd2e" candidate_uuid: "5fa951dba6ac49e398a13449b31694c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e58dbf1f0174d44955ed3d20569f53f"
I20260927 16:38:48.122752 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad903d9025f741fb80eb4c4b5820fd2e" candidate_uuid: "5fa951dba6ac49e398a13449b31694c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913c607885ed48c78f673a746ec084d8"
I20260927 16:38:48.122828 1665 raft_consensus.cc:3037] T ad903d9025f741fb80eb4c4b5820fd2e P 1e58dbf1f0174d44955ed3d20569f53f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:48.122869 1945 raft_consensus.cc:3037] T ad903d9025f741fb80eb4c4b5820fd2e P 913c607885ed48c78f673a746ec084d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:48.123876 1665 raft_consensus.cc:2454] T ad903d9025f741fb80eb4c4b5820fd2e P 1e58dbf1f0174d44955ed3d20569f53f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fa951dba6ac49e398a13449b31694c0 in term 1.
I20260927 16:38:48.124163 1945 raft_consensus.cc:2454] T ad903d9025f741fb80eb4c4b5820fd2e P 913c607885ed48c78f673a746ec084d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fa951dba6ac49e398a13449b31694c0 in term 1.
I20260927 16:38:48.124176 1739 leader_election.cc:302] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e58dbf1f0174d44955ed3d20569f53f, 5fa951dba6ac49e398a13449b31694c0; no voters:
I20260927 16:38:48.124356 2020 raft_consensus.cc:2781] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:38:48.124549 2020 raft_consensus.cc:686] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0 [term 1 LEADER]: Becoming Leader. State: Replica: 5fa951dba6ac49e398a13449b31694c0, State: Running, Role: LEADER
I20260927 16:38:48.124855 2020 consensus_queue.cc:237] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.128042 1506 catalog_manager.cc:5495] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fa951dba6ac49e398a13449b31694c0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "5fa951dba6ac49e398a13449b31694c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } health_report { overall_health: UNKNOWN } } }
I20260927 16:38:48.156358 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.153153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44110 (local address 127.0.98.2:34885)
0927 16:38:48.153277 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:48.153281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:48.153293 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:38:48.153440 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:48.153443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:48.153493 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:48.153575 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:38:48.153789 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.154273 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.155090 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.155359 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.155436 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.155691 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:38:48.156063 (+ 372us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:38:48.156143 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:48.156213 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 16:38:48.156258 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:38:48.161727 1665 raft_consensus.cc:1261] T ad903d9025f741fb80eb4c4b5820fd2e P 1e58dbf1f0174d44955ed3d20569f53f [term 1 FOLLOWER]: Refusing update from remote peer 5fa951dba6ac49e398a13449b31694c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:38:48.162050 1945 raft_consensus.cc:1261] T ad903d9025f741fb80eb4c4b5820fd2e P 913c607885ed48c78f673a746ec084d8 [term 1 FOLLOWER]: Refusing update from remote peer 5fa951dba6ac49e398a13449b31694c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:38:48.162407 2020 consensus_queue.cc:1035] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:38:48.162642 2024 consensus_queue.cc:1035] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:38:48.170336 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333999116943417344 to 7333999116801875968. Current Snapshot: MvccSnapshot[applied={T|T < 7333999116943417344}]
I20260927 16:38:48.180671 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:38:48.180830 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:38:48.183027 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb458acfde4b459aa26e0b6170f5c188: Sending DeleteTablet for 3 replicas of tablet ad903d9025f741fb80eb4c4b5820fd2e
I20260927 16:38:48.183775 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet ad903d9025f741fb80eb4c4b5820fd2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:38:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39524
I20260927 16:38:48.183787 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet ad903d9025f741fb80eb4c4b5820fd2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:38:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44104
I20260927 16:38:48.183890 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet ad903d9025f741fb80eb4c4b5820fd2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:38:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49052
I20260927 16:38:48.184252 2039 tablet_replica.cc:330] T ad903d9025f741fb80eb4c4b5820fd2e P 1e58dbf1f0174d44955ed3d20569f53f: stopping tablet replica
I20260927 16:38:48.184290 2038 tablet_replica.cc:330] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0: stopping tablet replica
I20260927 16:38:48.184548 2038 raft_consensus.cc:2229] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:38:48.184551 2039 raft_consensus.cc:2229] T ad903d9025f741fb80eb4c4b5820fd2e P 1e58dbf1f0174d44955ed3d20569f53f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:38:48.184693 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 16:38:48.184832 2039 raft_consensus.cc:2258] T ad903d9025f741fb80eb4c4b5820fd2e P 1e58dbf1f0174d44955ed3d20569f53f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:38:48.184863 2038 raft_consensus.cc:2258] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 16:38:48.185114 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:38:48.186301 2038 ts_tablet_manager.cc:1905] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:38:48.186359 2039 ts_tablet_manager.cc:1905] T ad903d9025f741fb80eb4c4b5820fd2e P 1e58dbf1f0174d44955ed3d20569f53f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:38:48.189110 2039 ts_tablet_manager.cc:1918] T ad903d9025f741fb80eb4c4b5820fd2e P 1e58dbf1f0174d44955ed3d20569f53f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:38:48.189206 2039 log.cc:1196] T ad903d9025f741fb80eb4c4b5820fd2e P 1e58dbf1f0174d44955ed3d20569f53f: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-0/wal/wals/ad903d9025f741fb80eb4c4b5820fd2e
I20260927 16:38:48.189215 2038 ts_tablet_manager.cc:1918] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:38:48.189291 2038 log.cc:1196] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-1/wal/wals/ad903d9025f741fb80eb4c4b5820fd2e
I20260927 16:38:48.189531 2039 ts_tablet_manager.cc:1939] T ad903d9025f741fb80eb4c4b5820fd2e P 1e58dbf1f0174d44955ed3d20569f53f: Deleting consensus metadata
I20260927 16:38:48.189590 2038 ts_tablet_manager.cc:1939] T ad903d9025f741fb80eb4c4b5820fd2e P 5fa951dba6ac49e398a13449b31694c0: Deleting consensus metadata
I20260927 16:38:48.190551 1495 catalog_manager.cc:4841] TS 5fa951dba6ac49e398a13449b31694c0 (127.0.98.2:34885): tablet ad903d9025f741fb80eb4c4b5820fd2e (table test-table [id=1935c23bf97a4f92a881df0a34ba6d4d]) successfully deleted
I20260927 16:38:48.190789 2037 tablet_replica.cc:330] T ad903d9025f741fb80eb4c4b5820fd2e P 913c607885ed48c78f673a746ec084d8: stopping tablet replica
I20260927 16:38:48.190855 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 68051e40e0fa4e5f8d37984344490e9e (DEFAULT_TABLE table=test-table [id=ef53329abe184868ba593fae87a71c66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:38:48.191078 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68051e40e0fa4e5f8d37984344490e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:48.191072 2037 raft_consensus.cc:2229] T ad903d9025f741fb80eb4c4b5820fd2e P 913c607885ed48c78f673a746ec084d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:38:48.191083 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 68051e40e0fa4e5f8d37984344490e9e (DEFAULT_TABLE table=test-table [id=ef53329abe184868ba593fae87a71c66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:38:48.191344 2037 raft_consensus.cc:2258] T ad903d9025f741fb80eb4c4b5820fd2e P 913c607885ed48c78f673a746ec084d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:38:48.191725 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68051e40e0fa4e5f8d37984344490e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:48.191751 1495 catalog_manager.cc:4841] TS 1e58dbf1f0174d44955ed3d20569f53f (127.0.98.1:42325): tablet ad903d9025f741fb80eb4c4b5820fd2e (table test-table [id=1935c23bf97a4f92a881df0a34ba6d4d]) successfully deleted
I20260927 16:38:48.193199 2016 tablet_bootstrap.cc:492] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0: Bootstrap starting.
I20260927 16:38:48.193324 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 68051e40e0fa4e5f8d37984344490e9e (DEFAULT_TABLE table=test-table [id=ef53329abe184868ba593fae87a71c66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:38:48.193553 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68051e40e0fa4e5f8d37984344490e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:48.193753 2015 tablet_bootstrap.cc:492] T 68051e40e0fa4e5f8d37984344490e9e P 1e58dbf1f0174d44955ed3d20569f53f: Bootstrap starting.
I20260927 16:38:48.194139 2016 tablet_bootstrap.cc:654] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:48.194716 2015 tablet_bootstrap.cc:654] T 68051e40e0fa4e5f8d37984344490e9e P 1e58dbf1f0174d44955ed3d20569f53f: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:48.196174 2016 tablet_bootstrap.cc:492] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0: No bootstrap required, opened a new log
I20260927 16:38:48.196174 2015 tablet_bootstrap.cc:492] T 68051e40e0fa4e5f8d37984344490e9e P 1e58dbf1f0174d44955ed3d20569f53f: No bootstrap required, opened a new log
I20260927 16:38:48.196262 2016 ts_tablet_manager.cc:1397] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 16:38:48.196262 2015 ts_tablet_manager.cc:1397] T 68051e40e0fa4e5f8d37984344490e9e P 1e58dbf1f0174d44955ed3d20569f53f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 16:38:48.196691 2015 raft_consensus.cc:348] T 68051e40e0fa4e5f8d37984344490e9e P 1e58dbf1f0174d44955ed3d20569f53f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.196691 2016 raft_consensus.cc:348] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.196800 2016 raft_consensus.cc:374] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:48.196800 2015 raft_consensus.cc:374] T 68051e40e0fa4e5f8d37984344490e9e P 1e58dbf1f0174d44955ed3d20569f53f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:48.196856 2015 raft_consensus.cc:729] T 68051e40e0fa4e5f8d37984344490e9e P 1e58dbf1f0174d44955ed3d20569f53f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e58dbf1f0174d44955ed3d20569f53f, State: Initialized, Role: FOLLOWER
I20260927 16:38:48.196856 2016 raft_consensus.cc:729] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa951dba6ac49e398a13449b31694c0, State: Initialized, Role: FOLLOWER
I20260927 16:38:48.196967 2016 consensus_queue.cc:260] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.196978 2015 consensus_queue.cc:260] T 68051e40e0fa4e5f8d37984344490e9e P 1e58dbf1f0174d44955ed3d20569f53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.197216 2016 ts_tablet_manager.cc:1428] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:38:48.197216 2015 ts_tablet_manager.cc:1428] T 68051e40e0fa4e5f8d37984344490e9e P 1e58dbf1f0174d44955ed3d20569f53f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:38:48.197414 2014 tablet_bootstrap.cc:492] T 68051e40e0fa4e5f8d37984344490e9e P 913c607885ed48c78f673a746ec084d8: Bootstrap starting.
I20260927 16:38:48.197455 2037 ts_tablet_manager.cc:1905] T ad903d9025f741fb80eb4c4b5820fd2e P 913c607885ed48c78f673a746ec084d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:38:48.198319 2014 tablet_bootstrap.cc:654] T 68051e40e0fa4e5f8d37984344490e9e P 913c607885ed48c78f673a746ec084d8: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:48.199940 2037 ts_tablet_manager.cc:1918] T ad903d9025f741fb80eb4c4b5820fd2e P 913c607885ed48c78f673a746ec084d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:38:48.200088 2037 log.cc:1196] T ad903d9025f741fb80eb4c4b5820fd2e P 913c607885ed48c78f673a746ec084d8: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-2/wal/wals/ad903d9025f741fb80eb4c4b5820fd2e
I20260927 16:38:48.200357 2014 tablet_bootstrap.cc:492] T 68051e40e0fa4e5f8d37984344490e9e P 913c607885ed48c78f673a746ec084d8: No bootstrap required, opened a new log
I20260927 16:38:48.200443 2037 ts_tablet_manager.cc:1939] T ad903d9025f741fb80eb4c4b5820fd2e P 913c607885ed48c78f673a746ec084d8: Deleting consensus metadata
I20260927 16:38:48.200467 2014 ts_tablet_manager.cc:1397] T 68051e40e0fa4e5f8d37984344490e9e P 913c607885ed48c78f673a746ec084d8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 16:38:48.200889 2014 raft_consensus.cc:348] T 68051e40e0fa4e5f8d37984344490e9e P 913c607885ed48c78f673a746ec084d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.200996 2014 raft_consensus.cc:374] T 68051e40e0fa4e5f8d37984344490e9e P 913c607885ed48c78f673a746ec084d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:48.201036 2014 raft_consensus.cc:729] T 68051e40e0fa4e5f8d37984344490e9e P 913c607885ed48c78f673a746ec084d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913c607885ed48c78f673a746ec084d8, State: Initialized, Role: FOLLOWER
I20260927 16:38:48.201169 2014 consensus_queue.cc:260] T 68051e40e0fa4e5f8d37984344490e9e P 913c607885ed48c78f673a746ec084d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.201470 2014 ts_tablet_manager.cc:1428] T 68051e40e0fa4e5f8d37984344490e9e P 913c607885ed48c78f673a746ec084d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260927 16:38:48.201545 1493 catalog_manager.cc:4841] TS 913c607885ed48c78f673a746ec084d8 (127.0.98.3:43199): tablet ad903d9025f741fb80eb4c4b5820fd2e (table test-table [id=1935c23bf97a4f92a881df0a34ba6d4d]) successfully deleted
W20260927 16:38:48.225683 1992 tablet.cc:2369] T 68051e40e0fa4e5f8d37984344490e9e P 913c607885ed48c78f673a746ec084d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:38:48.263744 1852 tablet.cc:2369] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:38:48.588795 2022 raft_consensus.cc:482] T 68051e40e0fa4e5f8d37984344490e9e P 1e58dbf1f0174d44955ed3d20569f53f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:38:48.588951 2022 raft_consensus.cc:504] T 68051e40e0fa4e5f8d37984344490e9e P 1e58dbf1f0174d44955ed3d20569f53f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.589421 2034 raft_consensus.cc:482] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:38:48.589564 2034 raft_consensus.cc:504] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.590337 2034 leader_election.cc:288] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e58dbf1f0174d44955ed3d20569f53f (127.0.98.1:42325), 913c607885ed48c78f673a746ec084d8 (127.0.98.3:43199)
I20260927 16:38:48.590637 2022 leader_election.cc:288] T 68051e40e0fa4e5f8d37984344490e9e P 1e58dbf1f0174d44955ed3d20569f53f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fa951dba6ac49e398a13449b31694c0 (127.0.98.2:34885), 913c607885ed48c78f673a746ec084d8 (127.0.98.3:43199)
I20260927 16:38:48.590761 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68051e40e0fa4e5f8d37984344490e9e" candidate_uuid: "5fa951dba6ac49e398a13449b31694c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e58dbf1f0174d44955ed3d20569f53f" is_pre_election: true
I20260927 16:38:48.590764 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68051e40e0fa4e5f8d37984344490e9e" candidate_uuid: "5fa951dba6ac49e398a13449b31694c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913c607885ed48c78f673a746ec084d8" is_pre_election: true
I20260927 16:38:48.590885 1665 raft_consensus.cc:2454] T 68051e40e0fa4e5f8d37984344490e9e P 1e58dbf1f0174d44955ed3d20569f53f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fa951dba6ac49e398a13449b31694c0 in term 0.
I20260927 16:38:48.590894 1945 raft_consensus.cc:2454] T 68051e40e0fa4e5f8d37984344490e9e P 913c607885ed48c78f673a746ec084d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fa951dba6ac49e398a13449b31694c0 in term 0.
I20260927 16:38:48.591185 1737 leader_election.cc:302] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fa951dba6ac49e398a13449b31694c0, 913c607885ed48c78f673a746ec084d8; no voters:
I20260927 16:38:48.591353 2034 raft_consensus.cc:2781] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:38:48.591403 2034 raft_consensus.cc:482] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:38:48.591437 2034 raft_consensus.cc:3037] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:48.592676 2034 raft_consensus.cc:504] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.593044 2034 leader_election.cc:288] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0 [CANDIDATE]: Term 1 election: Requested vote from peers 1e58dbf1f0174d44955ed3d20569f53f (127.0.98.1:42325), 913c607885ed48c78f673a746ec084d8 (127.0.98.3:43199)
I20260927 16:38:48.593325 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68051e40e0fa4e5f8d37984344490e9e" candidate_uuid: "5fa951dba6ac49e398a13449b31694c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e58dbf1f0174d44955ed3d20569f53f"
I20260927 16:38:48.593372 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68051e40e0fa4e5f8d37984344490e9e" candidate_uuid: "5fa951dba6ac49e398a13449b31694c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913c607885ed48c78f673a746ec084d8"
I20260927 16:38:48.593432 1665 raft_consensus.cc:3037] T 68051e40e0fa4e5f8d37984344490e9e P 1e58dbf1f0174d44955ed3d20569f53f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:48.593452 1945 raft_consensus.cc:3037] T 68051e40e0fa4e5f8d37984344490e9e P 913c607885ed48c78f673a746ec084d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:48.594461 1945 raft_consensus.cc:2454] T 68051e40e0fa4e5f8d37984344490e9e P 913c607885ed48c78f673a746ec084d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fa951dba6ac49e398a13449b31694c0 in term 1.
I20260927 16:38:48.594461 1665 raft_consensus.cc:2454] T 68051e40e0fa4e5f8d37984344490e9e P 1e58dbf1f0174d44955ed3d20569f53f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fa951dba6ac49e398a13449b31694c0 in term 1.
I20260927 16:38:48.594741 1737 leader_election.cc:302] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fa951dba6ac49e398a13449b31694c0, 913c607885ed48c78f673a746ec084d8; no voters:
I20260927 16:38:48.594923 2034 raft_consensus.cc:2781] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:38:48.595043 2034 raft_consensus.cc:686] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0 [term 1 LEADER]: Becoming Leader. State: Replica: 5fa951dba6ac49e398a13449b31694c0, State: Running, Role: LEADER
I20260927 16:38:48.595170 2034 consensus_queue.cc:237] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.595609 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.591145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53839 (local address 127.0.98.2:34885)
0927 16:38:48.591272 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:48.591276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:48.591357 (+ 81us) server_negotiation.cc:407] Connection header received
0927 16:38:48.591575 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:48.591580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:48.591634 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:48.591704 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:38:48.591924 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.592593 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.594300 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.594861 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.594935 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.595003 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:48.595436 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 16:38:48.595514 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 16:38:48.595629 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.590825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34885 (local address 127.0.98.1:53839)
0927 16:38:48.591267 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:48.591282 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:48.591436 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:48.591715 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:48.591717 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:48.591730 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:38:48.591841 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:48.591847 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.592734 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.592739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:48.594148 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:48.594158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.595084 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.595088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:48.595272 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.595327 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 16:38:48.595405 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 16:38:48.595496 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":140,"thread_start_us":76,"threads_started":1}
I20260927 16:38:48.595642 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.591252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44169 (local address 127.0.98.3:43199)
0927 16:38:48.591362 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:48.591366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:48.591591 (+ 225us) server_negotiation.cc:407] Connection header received
0927 16:38:48.591707 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:48.591709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:48.591747 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:48.591804 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:38:48.592104 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.592565 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.594300 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.594866 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.594935 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.594992 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:48.595492 (+ 500us) server_negotiation.cc:299] Negotiation successful
0927 16:38:48.595556 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 16:38:48.595959 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.590931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43199 (local address 127.0.98.1:44169)
0927 16:38:48.591523 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:48.591536 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:48.591613 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:48.591884 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:48.591889 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:48.591905 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:38:48.592000 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:48.592004 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.592674 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.592680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:48.594148 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:48.594158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.595084 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.595088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:48.595242 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.595370 (+ 128us) client_negotiation.cc:769] Sending connection context
0927 16:38:48.595780 (+ 410us) client_negotiation.cc:240] Negotiation successful
0927 16:38:48.595827 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":207,"thread_start_us":101,"threads_started":1}
I20260927 16:38:48.596282 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68051e40e0fa4e5f8d37984344490e9e" candidate_uuid: "1e58dbf1f0174d44955ed3d20569f53f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913c607885ed48c78f673a746ec084d8" is_pre_election: true
I20260927 16:38:48.596271 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68051e40e0fa4e5f8d37984344490e9e" candidate_uuid: "1e58dbf1f0174d44955ed3d20569f53f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa951dba6ac49e398a13449b31694c0" is_pre_election: true
I20260927 16:38:48.596421 1945 raft_consensus.cc:2379] T 68051e40e0fa4e5f8d37984344490e9e P 913c607885ed48c78f673a746ec084d8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e58dbf1f0174d44955ed3d20569f53f in current term 1: Already voted for candidate 5fa951dba6ac49e398a13449b31694c0 in this term.
I20260927 16:38:48.596874 1599 leader_election.cc:302] T 68051e40e0fa4e5f8d37984344490e9e P 1e58dbf1f0174d44955ed3d20569f53f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e58dbf1f0174d44955ed3d20569f53f; no voters: 5fa951dba6ac49e398a13449b31694c0, 913c607885ed48c78f673a746ec084d8
I20260927 16:38:48.597101 2022 raft_consensus.cc:2726] T 68051e40e0fa4e5f8d37984344490e9e P 1e58dbf1f0174d44955ed3d20569f53f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 16:38:48.597131 1506 catalog_manager.cc:5495] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fa951dba6ac49e398a13449b31694c0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "5fa951dba6ac49e398a13449b31694c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } health_report { overall_health: UNKNOWN } } }
Sep 27 16:38:48 dist-test-slave-z40b krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:38:48.668548 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.660730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36147 (local address 127.0.0.1:36442)
0927 16:38:48.661022 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:48.661035 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:48.661120 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:48.661350 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:48.661355 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:48.661735 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:48.661848 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:48.661858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.662598 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.662600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:48.663150 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:48.663157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.663605 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.663607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:48.663664 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.664163 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:48.664173 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:48.665472 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:48.667192 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:48.667233 (+ 41us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:48.667241 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:48.667452 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:48.667710 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:48.667712 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:48.667714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:48.667786 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:48.668091 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:48.668096 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:48.668185 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 16:38:48.668271 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 16:38:48.668368 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":111,"threads_started":1}
I20260927 16:38:48.668560 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.660762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36442 (local address 127.0.98.62:36147)
0927 16:38:48.661041 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:48.661048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:48.661079 (+ 31us) server_negotiation.cc:407] Connection header received
0927 16:38:48.661193 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:48.661195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:48.661229 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:48.661297 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:48.661948 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.662493 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.663314 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.663519 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.663589 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.665586 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:48.665589 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:48.665590 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:48.665609 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:48.667083 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:48.667561 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:48.667563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:48.667565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:48.667636 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:48.667859 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:48.667861 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:48.667862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:48.668007 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:48.668078 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:48.668323 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 16:38:48.668424 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":212,"thread_start_us":129,"threads_started":1}
I20260927 16:38:48.672150 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36442:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 16:38:48.672858 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:38:48.677299 1645 tablet_service.cc:1467] Processing CreateTablet for tablet ce60f69399fc430fbf8730f2d612214c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=53696fe49a034d69be1ff65fede63105]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 16:38:48.677340 1925 tablet_service.cc:1467] Processing CreateTablet for tablet ce60f69399fc430fbf8730f2d612214c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=53696fe49a034d69be1ff65fede63105]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 16:38:48.677317 1785 tablet_service.cc:1467] Processing CreateTablet for tablet ce60f69399fc430fbf8730f2d612214c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=53696fe49a034d69be1ff65fede63105]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 16:38:48.677675 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce60f69399fc430fbf8730f2d612214c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:48.677675 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce60f69399fc430fbf8730f2d612214c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:48.677726 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce60f69399fc430fbf8730f2d612214c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:48.680359 2015 tablet_bootstrap.cc:492] T ce60f69399fc430fbf8730f2d612214c P 1e58dbf1f0174d44955ed3d20569f53f: Bootstrap starting.
I20260927 16:38:48.680361 2016 tablet_bootstrap.cc:492] T ce60f69399fc430fbf8730f2d612214c P 5fa951dba6ac49e398a13449b31694c0: Bootstrap starting.
I20260927 16:38:48.680421 2014 tablet_bootstrap.cc:492] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8: Bootstrap starting.
I20260927 16:38:48.681128 2015 tablet_bootstrap.cc:654] T ce60f69399fc430fbf8730f2d612214c P 1e58dbf1f0174d44955ed3d20569f53f: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:48.681207 2016 tablet_bootstrap.cc:654] T ce60f69399fc430fbf8730f2d612214c P 5fa951dba6ac49e398a13449b31694c0: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:48.681207 2014 tablet_bootstrap.cc:654] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:48.682210 2015 tablet_bootstrap.cc:492] T ce60f69399fc430fbf8730f2d612214c P 1e58dbf1f0174d44955ed3d20569f53f: No bootstrap required, opened a new log
I20260927 16:38:48.682313 2015 ts_tablet_manager.cc:1397] T ce60f69399fc430fbf8730f2d612214c P 1e58dbf1f0174d44955ed3d20569f53f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 16:38:48.682435 2016 tablet_bootstrap.cc:492] T ce60f69399fc430fbf8730f2d612214c P 5fa951dba6ac49e398a13449b31694c0: No bootstrap required, opened a new log
I20260927 16:38:48.682513 2016 ts_tablet_manager.cc:1397] T ce60f69399fc430fbf8730f2d612214c P 5fa951dba6ac49e398a13449b31694c0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 16:38:48.682672 2014 tablet_bootstrap.cc:492] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8: No bootstrap required, opened a new log
I20260927 16:38:48.682745 2014 ts_tablet_manager.cc:1397] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 16:38:48.682763 2015 raft_consensus.cc:348] T ce60f69399fc430fbf8730f2d612214c P 1e58dbf1f0174d44955ed3d20569f53f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.682891 2015 raft_consensus.cc:374] T ce60f69399fc430fbf8730f2d612214c P 1e58dbf1f0174d44955ed3d20569f53f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:48.682880 2016 raft_consensus.cc:348] T ce60f69399fc430fbf8730f2d612214c P 5fa951dba6ac49e398a13449b31694c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.682926 2015 raft_consensus.cc:729] T ce60f69399fc430fbf8730f2d612214c P 1e58dbf1f0174d44955ed3d20569f53f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e58dbf1f0174d44955ed3d20569f53f, State: Initialized, Role: FOLLOWER
I20260927 16:38:48.682977 2016 raft_consensus.cc:374] T ce60f69399fc430fbf8730f2d612214c P 5fa951dba6ac49e398a13449b31694c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:48.683024 2016 raft_consensus.cc:729] T ce60f69399fc430fbf8730f2d612214c P 5fa951dba6ac49e398a13449b31694c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa951dba6ac49e398a13449b31694c0, State: Initialized, Role: FOLLOWER
I20260927 16:38:48.683051 2015 consensus_queue.cc:260] T ce60f69399fc430fbf8730f2d612214c P 1e58dbf1f0174d44955ed3d20569f53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.683125 2016 consensus_queue.cc:260] T ce60f69399fc430fbf8730f2d612214c P 5fa951dba6ac49e398a13449b31694c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.683136 2014 raft_consensus.cc:348] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.683257 2014 raft_consensus.cc:374] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:48.683295 2014 raft_consensus.cc:729] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913c607885ed48c78f673a746ec084d8, State: Initialized, Role: FOLLOWER
I20260927 16:38:48.683360 2015 ts_tablet_manager.cc:1428] T ce60f69399fc430fbf8730f2d612214c P 1e58dbf1f0174d44955ed3d20569f53f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:38:48.683423 2016 ts_tablet_manager.cc:1428] T ce60f69399fc430fbf8730f2d612214c P 5fa951dba6ac49e398a13449b31694c0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 16:38:48.683426 2014 consensus_queue.cc:260] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.683547 2022 tablet_replica.cc:439] T ce60f69399fc430fbf8730f2d612214c P 1e58dbf1f0174d44955ed3d20569f53f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } } }
I20260927 16:38:48.683653 2022 tablet_replica.cc:442] T ce60f69399fc430fbf8730f2d612214c P 1e58dbf1f0174d44955ed3d20569f53f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:38:48.683678 2014 ts_tablet_manager.cc:1428] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:38:48.683815 2020 tablet_replica.cc:439] T ce60f69399fc430fbf8730f2d612214c P 5fa951dba6ac49e398a13449b31694c0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } } }
I20260927 16:38:48.683835 2021 tablet_replica.cc:439] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } } }
I20260927 16:38:48.683926 2020 tablet_replica.cc:442] T ce60f69399fc430fbf8730f2d612214c P 5fa951dba6ac49e398a13449b31694c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:38:48.683948 2021 tablet_replica.cc:442] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:38:48.686690 2021 raft_consensus.cc:482] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:38:48.686759 2021 raft_consensus.cc:504] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.687760 2021 leader_election.cc:288] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e58dbf1f0174d44955ed3d20569f53f (127.0.98.1:42325), 5fa951dba6ac49e398a13449b31694c0 (127.0.98.2:34885)
I20260927 16:38:48.692030 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.688378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60325 (local address 127.0.98.2:34885)
0927 16:38:48.688501 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:48.688506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:48.688776 (+ 270us) server_negotiation.cc:407] Connection header received
0927 16:38:48.688928 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:48.688933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:48.688979 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:48.689053 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:38:48.689354 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.689920 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.690956 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.691403 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.691541 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.691595 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:48.691828 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 16:38:48.691919 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 16:38:48.692054 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.687943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42325 (local address 127.0.98.3:40505)
0927 16:38:48.688165 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:48.688180 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:48.688271 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:48.688683 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:48.688686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:48.688702 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:38:48.688829 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:48.688835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.689737 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.689741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:48.690766 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:48.690783 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.691541 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.691545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:48.691644 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.691699 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 16:38:48.691785 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 16:38:48.691900 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":49,"thread_start_us":95,"threads_started":1}
I20260927 16:38:48.692037 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.687993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40505 (local address 127.0.98.1:42325)
0927 16:38:48.688184 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:48.688187 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:48.688229 (+ 42us) server_negotiation.cc:407] Connection header received
0927 16:38:48.688528 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:48.688531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:48.688573 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:48.688665 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:38:48.688966 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.689617 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.690892 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.691400 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.691547 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.691602 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:48.691803 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 16:38:48.691897 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260927 16:38:48.692085 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.688296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34885 (local address 127.0.98.3:60325)
0927 16:38:48.688703 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:48.688716 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:48.688830 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:48.689142 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:48.689145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:48.689160 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:38:48.689252 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:48.689257 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.690029 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.690031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:48.690814 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:48.690822 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.691546 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.691550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:48.691654 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.691703 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 16:38:48.691809 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 16:38:48.691935 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":54,"thread_start_us":63,"threads_started":1}
I20260927 16:38:48.692605 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce60f69399fc430fbf8730f2d612214c" candidate_uuid: "913c607885ed48c78f673a746ec084d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e58dbf1f0174d44955ed3d20569f53f" is_pre_election: true
I20260927 16:38:48.692615 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce60f69399fc430fbf8730f2d612214c" candidate_uuid: "913c607885ed48c78f673a746ec084d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa951dba6ac49e398a13449b31694c0" is_pre_election: true
I20260927 16:38:48.692706 1665 raft_consensus.cc:2454] T ce60f69399fc430fbf8730f2d612214c P 1e58dbf1f0174d44955ed3d20569f53f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 913c607885ed48c78f673a746ec084d8 in term 0.
I20260927 16:38:48.692764 1805 raft_consensus.cc:2454] T ce60f69399fc430fbf8730f2d612214c P 5fa951dba6ac49e398a13449b31694c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 913c607885ed48c78f673a746ec084d8 in term 0.
I20260927 16:38:48.693096 1879 leader_election.cc:302] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e58dbf1f0174d44955ed3d20569f53f, 913c607885ed48c78f673a746ec084d8; no voters:
I20260927 16:38:48.693341 2021 raft_consensus.cc:2781] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:38:48.693408 2021 raft_consensus.cc:482] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:38:48.693457 2021 raft_consensus.cc:3037] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:48.694274 2021 raft_consensus.cc:504] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.694589 2021 leader_election.cc:288] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e58dbf1f0174d44955ed3d20569f53f (127.0.98.1:42325), 5fa951dba6ac49e398a13449b31694c0 (127.0.98.2:34885)
I20260927 16:38:48.694903 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce60f69399fc430fbf8730f2d612214c" candidate_uuid: "913c607885ed48c78f673a746ec084d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e58dbf1f0174d44955ed3d20569f53f"
I20260927 16:38:48.694963 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce60f69399fc430fbf8730f2d612214c" candidate_uuid: "913c607885ed48c78f673a746ec084d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa951dba6ac49e398a13449b31694c0"
I20260927 16:38:48.695035 1665 raft_consensus.cc:3037] T ce60f69399fc430fbf8730f2d612214c P 1e58dbf1f0174d44955ed3d20569f53f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:48.695087 1805 raft_consensus.cc:3037] T ce60f69399fc430fbf8730f2d612214c P 5fa951dba6ac49e398a13449b31694c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:48.695822 1665 raft_consensus.cc:2454] T ce60f69399fc430fbf8730f2d612214c P 1e58dbf1f0174d44955ed3d20569f53f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 913c607885ed48c78f673a746ec084d8 in term 1.
I20260927 16:38:48.695911 1805 raft_consensus.cc:2454] T ce60f69399fc430fbf8730f2d612214c P 5fa951dba6ac49e398a13449b31694c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 913c607885ed48c78f673a746ec084d8 in term 1.
I20260927 16:38:48.696146 1879 leader_election.cc:302] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e58dbf1f0174d44955ed3d20569f53f, 913c607885ed48c78f673a746ec084d8; no voters:
I20260927 16:38:48.696309 2021 raft_consensus.cc:2781] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:38:48.696463 2021 raft_consensus.cc:686] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8 [term 1 LEADER]: Becoming Leader. State: Replica: 913c607885ed48c78f673a746ec084d8, State: Running, Role: LEADER
I20260927 16:38:48.696687 2021 consensus_queue.cc:237] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } }
I20260927 16:38:48.697897 2059 tablet_replica.cc:439] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8: TxnStatusTablet state changed. Reason: New leader 913c607885ed48c78f673a746ec084d8. Latest consensus state: current_term: 1 leader_uuid: "913c607885ed48c78f673a746ec084d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } } }
I20260927 16:38:48.698015 2059 tablet_replica.cc:442] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8: This TxnStatusTablet replica's current role is: LEADER
I20260927 16:38:48.698239 2061 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 16:38:48.698375 2061 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 16:38:48.698873 1508 catalog_manager.cc:5495] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 913c607885ed48c78f673a746ec084d8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "913c607885ed48c78f673a746ec084d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } health_report { overall_health: HEALTHY } } }
I20260927 16:38:48.712219 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.708053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43199 (local address 127.0.0.1:39532)
0927 16:38:48.708280 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:48.708292 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:48.708375 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:48.708677 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:48.708679 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:48.708687 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:38:48.708765 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:48.708769 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.710033 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.710035 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:48.710613 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:48.710620 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.711152 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.711154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:48.711264 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.711305 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:38:48.711988 (+ 683us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:38:48.712001 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 16:38:48.712057 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 16:38:48.712122 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260927 16:38:48.712237 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.708149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39532 (local address 127.0.98.3:43199)
0927 16:38:48.708266 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:48.708271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:48.708330 (+ 59us) server_negotiation.cc:407] Connection header received
0927 16:38:48.708502 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:48.708506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:48.708561 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:48.708665 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:38:48.709200 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.709936 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.710717 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.711050 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.711150 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.711430 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:38:48.711894 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:38:48.712001 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:48.712071 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 16:38:48.712120 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:38:48.714674 1665 raft_consensus.cc:1261] T ce60f69399fc430fbf8730f2d612214c P 1e58dbf1f0174d44955ed3d20569f53f [term 1 FOLLOWER]: Refusing update from remote peer 913c607885ed48c78f673a746ec084d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:38:48.714811 1805 raft_consensus.cc:1261] T ce60f69399fc430fbf8730f2d612214c P 5fa951dba6ac49e398a13449b31694c0 [term 1 FOLLOWER]: Refusing update from remote peer 913c607885ed48c78f673a746ec084d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:38:48.715133 2059 consensus_queue.cc:1035] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:38:48.715296 2021 consensus_queue.cc:1035] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:38:48.716952 2020 tablet_replica.cc:439] T ce60f69399fc430fbf8730f2d612214c P 5fa951dba6ac49e398a13449b31694c0: TxnStatusTablet state changed. Reason: New leader 913c607885ed48c78f673a746ec084d8. Latest consensus state: current_term: 1 leader_uuid: "913c607885ed48c78f673a746ec084d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } } }
I20260927 16:38:48.717056 2020 tablet_replica.cc:442] T ce60f69399fc430fbf8730f2d612214c P 5fa951dba6ac49e398a13449b31694c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:38:48.716980 2022 tablet_replica.cc:439] T ce60f69399fc430fbf8730f2d612214c P 1e58dbf1f0174d44955ed3d20569f53f: TxnStatusTablet state changed. Reason: New leader 913c607885ed48c78f673a746ec084d8. Latest consensus state: current_term: 1 leader_uuid: "913c607885ed48c78f673a746ec084d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } } }
I20260927 16:38:48.717092 2022 tablet_replica.cc:442] T ce60f69399fc430fbf8730f2d612214c P 1e58dbf1f0174d44955ed3d20569f53f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:38:48.718839 2064 mvcc.cc:205] Tried to move back new op lower bound from 7333999119210823680 to 7333999119143890944. Current Snapshot: MvccSnapshot[applied={T|T < 7333999119210823680}]
I20260927 16:38:48.718940 2059 tablet_replica.cc:439] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "913c607885ed48c78f673a746ec084d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } } }
I20260927 16:38:48.719031 2059 tablet_replica.cc:442] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8: This TxnStatusTablet replica's current role is: LEADER
I20260927 16:38:48.719588 2020 tablet_replica.cc:439] T ce60f69399fc430fbf8730f2d612214c P 5fa951dba6ac49e398a13449b31694c0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "913c607885ed48c78f673a746ec084d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } } }
I20260927 16:38:48.719679 2020 tablet_replica.cc:442] T ce60f69399fc430fbf8730f2d612214c P 5fa951dba6ac49e398a13449b31694c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:38:48.719919 2021 tablet_replica.cc:439] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "913c607885ed48c78f673a746ec084d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } } }
I20260927 16:38:48.720037 2021 tablet_replica.cc:442] T ce60f69399fc430fbf8730f2d612214c P 913c607885ed48c78f673a746ec084d8: This TxnStatusTablet replica's current role is: LEADER
I20260927 16:38:48.720889 2022 tablet_replica.cc:439] T ce60f69399fc430fbf8730f2d612214c P 1e58dbf1f0174d44955ed3d20569f53f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "913c607885ed48c78f673a746ec084d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 } } peers { permanent_uuid: "5fa951dba6ac49e398a13449b31694c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34885 } } peers { permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 } } }
I20260927 16:38:48.720981 2022 tablet_replica.cc:442] T ce60f69399fc430fbf8730f2d612214c P 1e58dbf1f0174d44955ed3d20569f53f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:38:48.723650 2067 mvcc.cc:205] Tried to move back new op lower bound from 7333999119210823680 to 7333999119143890944. Current Snapshot: MvccSnapshot[applied={T|T < 7333999119210823680}]
I20260927 16:38:48.734917 2075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.731543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43199 (local address 127.0.98.2:35225)
0927 16:38:48.731783 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:48.731796 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:48.731878 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:48.732082 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:48.732084 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:48.732090 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:38:48.732167 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:48.732171 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.733096 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.733098 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:48.733748 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:48.733757 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.734294 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.734297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:48.734356 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.734412 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:38:48.734693 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:38:48.734698 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 16:38:48.734746 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 16:38:48.734809 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":91,"threads_started":1}
I20260927 16:38:48.734911 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.731601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35225 (local address 127.0.98.3:43199)
0927 16:38:48.731697 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:48.731701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:48.731863 (+ 162us) server_negotiation.cc:407] Connection header received
0927 16:38:48.731946 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:48.731949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:48.731993 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:48.732069 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:38:48.732257 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.733016 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.733875 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.734199 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.734295 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.734546 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:38:48.734617 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:38:48.734684 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:48.734755 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 16:38:48.734799 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 16:38:48.740532 1945 raft_consensus.cc:1261] T 68051e40e0fa4e5f8d37984344490e9e P 913c607885ed48c78f673a746ec084d8 [term 1 FOLLOWER]: Refusing update from remote peer 5fa951dba6ac49e398a13449b31694c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:38:48.740824 1665 raft_consensus.cc:1261] T 68051e40e0fa4e5f8d37984344490e9e P 1e58dbf1f0174d44955ed3d20569f53f [term 1 FOLLOWER]: Refusing update from remote peer 5fa951dba6ac49e398a13449b31694c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:38:48.740883 2034 consensus_queue.cc:1035] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "913c607885ed48c78f673a746ec084d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:38:48.741184 2034 consensus_queue.cc:1035] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e58dbf1f0174d44955ed3d20569f53f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:38:48.763613 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.759295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38245 (local address 127.0.98.2:34885)
0927 16:38:48.759406 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:48.759410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:48.760035 (+ 625us) server_negotiation.cc:407] Connection header received
0927 16:38:48.760136 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:48.760138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:48.760173 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:48.760231 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:38:48.760500 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.761176 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.762498 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.762824 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.762929 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.763194 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:38:48.763282 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:38:48.763345 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:48.763441 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 16:38:48.763501 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 16:38:48.763659 2079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.759230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34885 (local address 127.0.98.3:38245)
0927 16:38:48.759944 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:48.759958 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:48.760058 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:48.760252 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:48.760255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:48.760263 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:38:48.760391 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:48.760397 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.761273 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.761276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:48.762387 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:48.762397 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.762938 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.762942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:48.763030 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.763076 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:38:48.763364 (+ 288us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:38:48.763370 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 16:38:48.763434 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 16:38:48.763510 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":108,"threads_started":1}
I20260927 16:38:48.784368 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:38:48.784494 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:38:48.786530 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb458acfde4b459aa26e0b6170f5c188: Sending DeleteTablet for 3 replicas of tablet 68051e40e0fa4e5f8d37984344490e9e
I20260927 16:38:48.787075 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 68051e40e0fa4e5f8d37984344490e9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:38:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49052
I20260927 16:38:48.787175 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 68051e40e0fa4e5f8d37984344490e9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:38:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39524
I20260927 16:38:48.787569 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 68051e40e0fa4e5f8d37984344490e9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:38:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44104
I20260927 16:38:48.787896 2083 tablet_replica.cc:330] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0: stopping tablet replica
I20260927 16:38:48.788043 2083 raft_consensus.cc:2229] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:38:48.788203 2083 raft_consensus.cc:2258] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:38:48.788415 2082 tablet_replica.cc:330] T 68051e40e0fa4e5f8d37984344490e9e P 913c607885ed48c78f673a746ec084d8: stopping tablet replica
I20260927 16:38:48.788916 2083 ts_tablet_manager.cc:1905] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:38:48.791399 2081 tablet_replica.cc:330] T 68051e40e0fa4e5f8d37984344490e9e P 1e58dbf1f0174d44955ed3d20569f53f: stopping tablet replica
I20260927 16:38:48.791531 2081 raft_consensus.cc:2229] T 68051e40e0fa4e5f8d37984344490e9e P 1e58dbf1f0174d44955ed3d20569f53f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:38:48.791636 2083 ts_tablet_manager.cc:1918] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:38:48.791659 2081 raft_consensus.cc:2258] T 68051e40e0fa4e5f8d37984344490e9e P 1e58dbf1f0174d44955ed3d20569f53f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:38:48.791709 2083 log.cc:1196] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-1/wal/wals/68051e40e0fa4e5f8d37984344490e9e
I20260927 16:38:48.792024 2083 ts_tablet_manager.cc:1939] T 68051e40e0fa4e5f8d37984344490e9e P 5fa951dba6ac49e398a13449b31694c0: Deleting consensus metadata
I20260927 16:38:48.792265 2081 ts_tablet_manager.cc:1905] T 68051e40e0fa4e5f8d37984344490e9e P 1e58dbf1f0174d44955ed3d20569f53f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:38:48.792572 2082 raft_consensus.cc:2229] T 68051e40e0fa4e5f8d37984344490e9e P 913c607885ed48c78f673a746ec084d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:38:48.792649 1495 catalog_manager.cc:4841] TS 5fa951dba6ac49e398a13449b31694c0 (127.0.98.2:34885): tablet 68051e40e0fa4e5f8d37984344490e9e (table test-table [id=ef53329abe184868ba593fae87a71c66]) successfully deleted
I20260927 16:38:48.792694 2082 raft_consensus.cc:2258] T 68051e40e0fa4e5f8d37984344490e9e P 913c607885ed48c78f673a746ec084d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:38:48.793262 2082 ts_tablet_manager.cc:1905] T 68051e40e0fa4e5f8d37984344490e9e P 913c607885ed48c78f673a746ec084d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:38:48.794523 2081 ts_tablet_manager.cc:1918] T 68051e40e0fa4e5f8d37984344490e9e P 1e58dbf1f0174d44955ed3d20569f53f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:38:48.794620 2081 log.cc:1196] T 68051e40e0fa4e5f8d37984344490e9e P 1e58dbf1f0174d44955ed3d20569f53f: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-0/wal/wals/68051e40e0fa4e5f8d37984344490e9e
I20260927 16:38:48.795148 2081 ts_tablet_manager.cc:1939] T 68051e40e0fa4e5f8d37984344490e9e P 1e58dbf1f0174d44955ed3d20569f53f: Deleting consensus metadata
I20260927 16:38:48.795729 1495 catalog_manager.cc:4841] TS 1e58dbf1f0174d44955ed3d20569f53f (127.0.98.1:42325): tablet 68051e40e0fa4e5f8d37984344490e9e (table test-table [id=ef53329abe184868ba593fae87a71c66]) successfully deleted
I20260927 16:38:48.796314 2082 ts_tablet_manager.cc:1918] T 68051e40e0fa4e5f8d37984344490e9e P 913c607885ed48c78f673a746ec084d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:38:48.796387 2082 log.cc:1196] T 68051e40e0fa4e5f8d37984344490e9e P 913c607885ed48c78f673a746ec084d8: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790527121515808-392-0/minicluster-data/ts-2/wal/wals/68051e40e0fa4e5f8d37984344490e9e
I20260927 16:38:48.796651 2082 ts_tablet_manager.cc:1939] T 68051e40e0fa4e5f8d37984344490e9e P 913c607885ed48c78f673a746ec084d8: Deleting consensus metadata
I20260927 16:38:48.797220 1493 catalog_manager.cc:4841] TS 913c607885ed48c78f673a746ec084d8 (127.0.98.3:43199): tablet 68051e40e0fa4e5f8d37984344490e9e (table test-table [id=ef53329abe184868ba593fae87a71c66]) successfully deleted
Sep 27 16:38:48 dist-test-slave-z40b krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527127, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 16:38:48.807555 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.797410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49066 (local address 127.0.98.1:42325)
0927 16:38:48.797528 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:48.797538 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:48.797746 (+ 208us) server_negotiation.cc:407] Connection header received
0927 16:38:48.797984 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:48.797990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:48.798041 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:48.798127 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:48.798910 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.799437 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.800557 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.800755 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.800821 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.803888 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:48.803902 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:48.803913 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:48.803933 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:48.805735 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:48.806254 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:48.806260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:48.806261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:48.806332 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:48.806621 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:48.806625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:48.806626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:48.806851 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:48.807049 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:48.807281 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 16:38:48.807395 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 16:38:48.807942 1705 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49066
I20260927 16:38:48.816917 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:48.809513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36446 (local address 127.0.98.62:36147)
0927 16:38:48.809628 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:48.809631 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:48.809680 (+ 49us) server_negotiation.cc:407] Connection header received
0927 16:38:48.809856 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:48.809859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:48.809907 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:48.809987 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:48.810714 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.811260 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.812309 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:48.812508 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:48.812579 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:48.813953 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:48.813956 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:48.813958 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:48.813980 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:48.815424 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:48.815843 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:48.815845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:48.815846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:48.815912 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:48.816163 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:48.816165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:48.816166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:48.816307 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:48.816401 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:48.816666 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 16:38:48.816755 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 16:38:48.817314 1507 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36446
I20260927 16:38:48.818394 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 1580
I20260927 16:38:48.826411 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 1720
I20260927 16:38:48.837155 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 1860
I20260927 16:38:48.843420 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 1476
2026-09-27T16:38:48Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2868 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:38:48 dist-test-slave-z40b krb5kdc[2090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:38:48 dist-test-slave-z40b krb5kdc[2090](info): set up 2 sockets
Sep 27 16:38:48 dist-test-slave-z40b krb5kdc[2090](info): commencing operation
krb5kdc: starting...
W20260927 16:38:49.834769 392 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:38:49 dist-test-slave-z40b krb5kdc[2090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:38:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:38:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:38:49.945744 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44319
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:38:50.045589 2103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:38:50.045817 2103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:38:50.045855 2103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:38:50.049727 2103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:38:50.049770 2103 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 16:38:50.049785 2103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:38:50.049801 2103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:38:50.049865 2103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:38:50.049885 2103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:38:50.054346 2103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44319
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:38:50.055315 2103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:38:50.056214 2103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:38:50.064126 2109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:50.064175 2111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:50.064131 2108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:50.064196 2103 server_base.cc:1034] running on GCE node
I20260927 16:38:50.064859 2103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:38:50.065807 2103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:38:50.066989 2103 hybrid_clock.cc:648] HybridClock initialized: now 1790527130066973 us; error 37 us; skew 500 ppm
Sep 27 16:38:50 dist-test-slave-z40b krb5kdc[2090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:38:50.069644 2103 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:38:50.070403 2103 webserver.cc:460] Webserver started at http://127.0.98.62:34805/ using document root <none> and password file <none>
I20260927 16:38:50.070852 2103 fs_manager.cc:359] Metadata directory not provided
I20260927 16:38:50.070905 2103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:38:50.071044 2103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:38:50.072817 2103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "2cc203aec9eb4fe4b7a505cacdf36c3f"
format_stamp: "Formatted at 2026-09-27 16:38:50 on dist-test-slave-z40b"
server_key: "7f07c3243dfd90a1681cd9ee72172f8a"
server_key_iv: "915307e8cf5ec6d9a20dae1bfa2973ac"
server_key_version: "encryptionkey@0"
I20260927 16:38:50.073271 2103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "2cc203aec9eb4fe4b7a505cacdf36c3f"
format_stamp: "Formatted at 2026-09-27 16:38:50 on dist-test-slave-z40b"
server_key: "7f07c3243dfd90a1681cd9ee72172f8a"
server_key_iv: "915307e8cf5ec6d9a20dae1bfa2973ac"
server_key_version: "encryptionkey@0"
I20260927 16:38:50.076562 2103 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 16:38:50.078675 2118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:50.079830 2103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:38:50.079962 2103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "2cc203aec9eb4fe4b7a505cacdf36c3f"
format_stamp: "Formatted at 2026-09-27 16:38:50 on dist-test-slave-z40b"
server_key: "7f07c3243dfd90a1681cd9ee72172f8a"
server_key_iv: "915307e8cf5ec6d9a20dae1bfa2973ac"
server_key_version: "encryptionkey@0"
I20260927 16:38:50.080049 2103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:38:50.092612 2103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:38:50.095009 2103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:38:50.095141 2103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:38:50.102499 2103 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44319
I20260927 16:38:50.102523 2180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44319 every 8 connection(s)
I20260927 16:38:50.103400 2103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:38:50.105971 2181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:50.110576 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 2103
I20260927 16:38:50.110704 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:38:50.110947 392 external_mini_cluster.cc:1428] Setting key 552de90e17d7ba8b4236f3c4583d05a0
I20260927 16:38:50.111156 2181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cc203aec9eb4fe4b7a505cacdf36c3f: Bootstrap starting.
I20260927 16:38:50.113471 2181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cc203aec9eb4fe4b7a505cacdf36c3f: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:50.114228 2181 log.cc:824] T 00000000000000000000000000000000 P 2cc203aec9eb4fe4b7a505cacdf36c3f: Log is configured to *not* fsync() on all Append() calls
I20260927 16:38:50.116329 2181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cc203aec9eb4fe4b7a505cacdf36c3f: No bootstrap required, opened a new log
Sep 27 16:38:50 dist-test-slave-z40b krb5kdc[2090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:38:50.120288 2181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cc203aec9eb4fe4b7a505cacdf36c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cc203aec9eb4fe4b7a505cacdf36c3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44319 } }
I20260927 16:38:50.120539 2181 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cc203aec9eb4fe4b7a505cacdf36c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:50.120602 2181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cc203aec9eb4fe4b7a505cacdf36c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cc203aec9eb4fe4b7a505cacdf36c3f, State: Initialized, Role: FOLLOWER
I20260927 16:38:50.121062 2181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cc203aec9eb4fe4b7a505cacdf36c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cc203aec9eb4fe4b7a505cacdf36c3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44319 } }
I20260927 16:38:50.121224 2181 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cc203aec9eb4fe4b7a505cacdf36c3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:38:50.121289 2181 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cc203aec9eb4fe4b7a505cacdf36c3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:38:50.121390 2181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cc203aec9eb4fe4b7a505cacdf36c3f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:50.122421 2181 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cc203aec9eb4fe4b7a505cacdf36c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cc203aec9eb4fe4b7a505cacdf36c3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44319 } }
I20260927 16:38:50.122759 2181 leader_election.cc:302] T 00000000000000000000000000000000 P 2cc203aec9eb4fe4b7a505cacdf36c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cc203aec9eb4fe4b7a505cacdf36c3f; no voters:
I20260927 16:38:50.123032 2181 leader_election.cc:288] T 00000000000000000000000000000000 P 2cc203aec9eb4fe4b7a505cacdf36c3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:38:50.123117 2186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cc203aec9eb4fe4b7a505cacdf36c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:38:50.123363 2186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cc203aec9eb4fe4b7a505cacdf36c3f [term 1 LEADER]: Becoming Leader. State: Replica: 2cc203aec9eb4fe4b7a505cacdf36c3f, State: Running, Role: LEADER
I20260927 16:38:50.123638 2186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cc203aec9eb4fe4b7a505cacdf36c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cc203aec9eb4fe4b7a505cacdf36c3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44319 } }
I20260927 16:38:50.124287 2181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cc203aec9eb4fe4b7a505cacdf36c3f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:38:50.124595 2186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cc203aec9eb4fe4b7a505cacdf36c3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cc203aec9eb4fe4b7a505cacdf36c3f. Latest consensus state: current_term: 1 leader_uuid: "2cc203aec9eb4fe4b7a505cacdf36c3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cc203aec9eb4fe4b7a505cacdf36c3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44319 } } }
I20260927 16:38:50.124703 2186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cc203aec9eb4fe4b7a505cacdf36c3f [sys.catalog]: This master's current role is: LEADER
I20260927 16:38:50.125103 2188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cc203aec9eb4fe4b7a505cacdf36c3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cc203aec9eb4fe4b7a505cacdf36c3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cc203aec9eb4fe4b7a505cacdf36c3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44319 } } }
I20260927 16:38:50.125505 2188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cc203aec9eb4fe4b7a505cacdf36c3f [sys.catalog]: This master's current role is: LEADER
I20260927 16:38:50.126441 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:50.112234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53764 (local address 127.0.98.62:44319)
0927 16:38:50.112642 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:50.112656 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:50.112688 (+ 32us) server_negotiation.cc:407] Connection header received
0927 16:38:50.113277 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:50.113293 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:50.113645 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:50.113990 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:50.114862 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.116236 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.117083 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.117836 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.117983 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:50.120521 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:50.120526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:50.120551 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:50.120585 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:50.122561 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:50.123011 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:50.123018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:50.123024 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:50.123241 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:50.123567 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:50.123570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:50.123572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:50.124024 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:50.124209 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:50.125160 (+ 951us) server_negotiation.cc:299] Negotiation successful
0927 16:38:50.125443 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":136,"threads_started":1}
I20260927 16:38:50.126789 2190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:38:50.129590 2190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:38:50.134550 2190 catalog_manager.cc:1347] Generated new cluster ID: 943db20e8522458f8e5f1f8246343f2f
I20260927 16:38:50.134608 2190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:38:50.142772 2190 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:38:50.143577 2190 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:38:50.152503 2190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cc203aec9eb4fe4b7a505cacdf36c3f: Generated new TSK 0
I20260927 16:38:50.153126 2190 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:38:50.178954 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44319
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:38:50.282123 2207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:38:50.282349 2207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:38:50.282433 2207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:38:50.286499 2207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:38:50.286551 2207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:38:50.286623 2207 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 16:38:50.286654 2207 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 16:38:50.286733 2207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:38:50.292277 2207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:38:50.293159 2207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:38:50.294068 2207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:38:50.302899 2212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:50.302870 2215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:50.302891 2207 server_base.cc:1034] running on GCE node
W20260927 16:38:50.302909 2213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:50.303452 2207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:38:50.304018 2207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:38:50.305166 2207 hybrid_clock.cc:648] HybridClock initialized: now 1790527130305150 us; error 28 us; skew 500 ppm
Sep 27 16:38:50 dist-test-slave-z40b krb5kdc[2090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:38:50.307797 2207 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:38:50.308552 2207 webserver.cc:460] Webserver started at http://127.0.98.1:39155/ using document root <none> and password file <none>
I20260927 16:38:50.309074 2207 fs_manager.cc:359] Metadata directory not provided
I20260927 16:38:50.309165 2207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:38:50.309370 2207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:38:50.311129 2207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-0/data/instance:
uuid: "b8c2d915da9f49abbac90af10ecd3337"
format_stamp: "Formatted at 2026-09-27 16:38:50 on dist-test-slave-z40b"
server_key: "137a4e8b34dafe33ab73dfed46c96b18"
server_key_iv: "f2f4d65cf23a6534d9443bcfe8a7db03"
server_key_version: "encryptionkey@0"
I20260927 16:38:50.311700 2207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b8c2d915da9f49abbac90af10ecd3337"
format_stamp: "Formatted at 2026-09-27 16:38:50 on dist-test-slave-z40b"
server_key: "137a4e8b34dafe33ab73dfed46c96b18"
server_key_iv: "f2f4d65cf23a6534d9443bcfe8a7db03"
server_key_version: "encryptionkey@0"
I20260927 16:38:50.314997 2207 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:38:50.317188 2222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:50.318342 2207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:38:50.318470 2207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "b8c2d915da9f49abbac90af10ecd3337"
format_stamp: "Formatted at 2026-09-27 16:38:50 on dist-test-slave-z40b"
server_key: "137a4e8b34dafe33ab73dfed46c96b18"
server_key_iv: "f2f4d65cf23a6534d9443bcfe8a7db03"
server_key_version: "encryptionkey@0"
I20260927 16:38:50.318572 2207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:38:50.332054 2207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:38:50.335094 2207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:38:50.335306 2207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:38:50.336584 2207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:38:50.336653 2207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:50.336715 2207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:38:50.336772 2207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 16:38:50 dist-test-slave-z40b krb5kdc[2090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:38:50.347312 2207 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36215
I20260927 16:38:50.347534 2337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36215 every 8 connection(s)
I20260927 16:38:50.348222 2207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:38:50.350026 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:50.337362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55203 (local address 127.0.98.62:44319)
0927 16:38:50.337503 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:50.337508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:50.338290 (+ 782us) server_negotiation.cc:407] Connection header received
0927 16:38:50.339460 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:50.339464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:50.339517 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:50.339616 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:50.340956 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.341520 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.342813 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.343238 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.343352 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:50.346380 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:50.346383 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:50.346386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:50.346410 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:50.348028 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:50.348601 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:50.348605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:50.348607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:50.348696 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:50.349058 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:50.349063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:50.349068 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:50.349237 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:50.349338 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:50.349767 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 16:38:50.349874 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 16:38:50.350700 2231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:50.337700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44319 (local address 127.0.98.1:55203)
0927 16:38:50.338126 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:50.338171 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:50.339180 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:50.339776 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:50.339786 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:50.340225 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:50.340815 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:50.340835 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.341644 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.341647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:50.342686 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:50.342695 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.343402 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.343410 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:50.343646 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:50.344318 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:50.344336 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:50.346156 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:50.348157 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:50.348163 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:50.348175 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:50.348477 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:50.348800 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:50.348803 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:50.348805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:50.348967 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:50.349336 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:50.349341 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:50.349550 (+ 209us) client_negotiation.cc:769] Sending connection context
0927 16:38:50.349716 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 16:38:50.349984 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":117,"threads_started":1}
I20260927 16:38:50.351927 2338 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44319
I20260927 16:38:50.352077 2338 heartbeater.cc:464] Registering TS with master...
I20260927 16:38:50.352501 2338 heartbeater.cc:511] Master 127.0.98.62:44319 requested a full tablet report, sending...
I20260927 16:38:50.353869 2134 ts_manager.cc:194] Registered new tserver with Master: b8c2d915da9f49abbac90af10ecd3337 (127.0.98.1:36215)
I20260927 16:38:50.354255 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 2207
I20260927 16:38:50.354379 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:38:50.354663 392 external_mini_cluster.cc:1428] Setting key 395064a11ef0d4198159f5c76ce34132
I20260927 16:38:50.355170 2134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55203
I20260927 16:38:50.360832 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:50.353770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40443 (local address 127.0.98.62:44319)
0927 16:38:50.353878 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:50.353882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:50.354112 (+ 230us) server_negotiation.cc:407] Connection header received
0927 16:38:50.354212 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:50.354215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:50.354247 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:50.354320 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:50.354793 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.355428 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.356468 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.356715 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.356771 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:50.357871 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:50.357877 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:50.357882 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:50.357913 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:50.359280 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:50.359712 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:50.359716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:50.359718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:50.359794 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:50.360110 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:50.360115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:50.360116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:50.360269 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:50.360366 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:50.360606 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 16:38:50.360697 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:38:50.360841 2344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:50.353712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44319 (local address 127.0.98.1:40443)
0927 16:38:50.354018 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:50.354033 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:50.354139 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:50.354351 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:50.354354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:50.354529 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:50.354645 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:50.354652 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.355553 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.355556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:50.356342 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:50.356353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.356889 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.356892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:50.356964 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:50.357395 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:50.357414 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:50.357710 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:50.359403 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:50.359406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:50.359408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:50.359612 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:50.359890 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:50.359894 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:50.359896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:50.359975 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:50.360358 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:50.360361 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:50.360441 (+ 80us) client_negotiation.cc:769] Sending connection context
0927 16:38:50.360540 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 16:38:50.360664 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:38:50.382282 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44319
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:38:50.483999 2347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:38:50.484234 2347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:38:50.484277 2347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:38:50.488008 2347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:38:50.488050 2347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:38:50.488087 2347 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 16:38:50.488103 2347 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 16:38:50.488134 2347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:38:50.492993 2347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:38:50.493780 2347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:38:50.494573 2347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:38:50.503468 2355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:50.503502 2347 server_base.cc:1034] running on GCE node
W20260927 16:38:50.503472 2353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:50.503535 2352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:50.504068 2347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:38:50.504626 2347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:38:50.505806 2347 hybrid_clock.cc:648] HybridClock initialized: now 1790527130505779 us; error 44 us; skew 500 ppm
Sep 27 16:38:50 dist-test-slave-z40b krb5kdc[2090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:38:50.508504 2347 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:38:50.509306 2347 webserver.cc:460] Webserver started at http://127.0.98.2:33773/ using document root <none> and password file <none>
I20260927 16:38:50.509807 2347 fs_manager.cc:359] Metadata directory not provided
I20260927 16:38:50.509888 2347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:38:50.510095 2347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:38:50.512008 2347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-1/data/instance:
uuid: "6de1875303d446e1841ca078453646ef"
format_stamp: "Formatted at 2026-09-27 16:38:50 on dist-test-slave-z40b"
server_key: "97eca3a6aa44749866299cd1aac2f80e"
server_key_iv: "8b406b7fc7fb297e11ffab984ae1a871"
server_key_version: "encryptionkey@0"
I20260927 16:38:50.512506 2347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6de1875303d446e1841ca078453646ef"
format_stamp: "Formatted at 2026-09-27 16:38:50 on dist-test-slave-z40b"
server_key: "97eca3a6aa44749866299cd1aac2f80e"
server_key_iv: "8b406b7fc7fb297e11ffab984ae1a871"
server_key_version: "encryptionkey@0"
I20260927 16:38:50.515805 2347 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:38:50.517849 2362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:50.518920 2347 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:38:50.519071 2347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "6de1875303d446e1841ca078453646ef"
format_stamp: "Formatted at 2026-09-27 16:38:50 on dist-test-slave-z40b"
server_key: "97eca3a6aa44749866299cd1aac2f80e"
server_key_iv: "8b406b7fc7fb297e11ffab984ae1a871"
server_key_version: "encryptionkey@0"
I20260927 16:38:50.519165 2347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:38:50.540645 2347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:38:50.543174 2347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:38:50.543367 2347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:38:50.544587 2347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:38:50.544653 2347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:50.544718 2347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:38:50.544759 2347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 16:38:50 dist-test-slave-z40b krb5kdc[2090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:38:50.554760 2347 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38761
I20260927 16:38:50.554782 2477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38761 every 8 connection(s)
I20260927 16:38:50.555611 2347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:38:50.557317 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:50.545385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53055 (local address 127.0.98.62:44319)
0927 16:38:50.545556 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:50.545563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:50.546338 (+ 775us) server_negotiation.cc:407] Connection header received
0927 16:38:50.547388 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:50.547392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:50.547450 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:50.547508 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:50.549060 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.549642 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.550753 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.550953 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.551076 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:50.553849 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:50.553852 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:50.553855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:50.553878 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:50.555383 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:50.555882 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:50.555886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:50.555888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:50.555969 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:50.556330 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:50.556333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:50.556335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:50.556540 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:50.556637 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:50.557074 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 16:38:50.557170 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 16:38:50.557482 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 2347
I20260927 16:38:50.557605 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:38:50.557883 392 external_mini_cluster.cc:1428] Setting key bdc6898c806e5eb24c03b6fb80e8d224
I20260927 16:38:50.558060 2371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:50.545718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44319 (local address 127.0.98.2:53055)
0927 16:38:50.546142 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:50.546213 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:50.547135 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:50.547875 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:50.547881 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:50.548277 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:50.548848 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:50.548867 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.549758 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.549761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:50.550635 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:50.550643 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.551124 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.551131 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:50.551367 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:50.551964 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:50.551985 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:50.553702 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:50.555484 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:50.555489 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:50.555502 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:50.555799 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:50.556057 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:50.556062 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:50.556064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:50.556196 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:50.556653 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:50.556658 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:50.556874 (+ 216us) client_negotiation.cc:769] Sending connection context
0927 16:38:50.557021 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 16:38:50.557231 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":121,"threads_started":1}
I20260927 16:38:50.559194 2478 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44319
I20260927 16:38:50.559341 2478 heartbeater.cc:464] Registering TS with master...
I20260927 16:38:50.559831 2478 heartbeater.cc:511] Master 127.0.98.62:44319 requested a full tablet report, sending...
I20260927 16:38:50.560815 2134 ts_manager.cc:194] Registered new tserver with Master: 6de1875303d446e1841ca078453646ef (127.0.98.2:38761)
I20260927 16:38:50.561480 2134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53055
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 16:38:50.568747 2485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:50.561222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44319 (local address 127.0.98.2:59283)
0927 16:38:50.561556 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:50.561572 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:50.561675 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:50.561904 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:50.561906 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:50.562129 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:50.562231 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:50.562237 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.563186 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.563193 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:50.564220 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:50.564233 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.564638 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.564644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:50.564774 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:50.565230 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:50.565248 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:50.565705 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:50.567346 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:50.567352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:50.567355 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:50.567596 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:50.567868 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:50.567871 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:50.567873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:50.567972 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:50.568360 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:50.568362 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:50.568432 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 16:38:50.568516 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 16:38:50.568612 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":115,"threads_started":1}
I20260927 16:38:50.568751 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:50.561280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59283 (local address 127.0.98.62:44319)
0927 16:38:50.561365 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:50.561369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:50.561661 (+ 292us) server_negotiation.cc:407] Connection header received
0927 16:38:50.561749 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:50.561752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:50.561794 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:50.561867 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:50.562365 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.563034 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.564335 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.564525 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.564601 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:50.565830 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:50.565833 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:50.565835 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:50.565854 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:50.567191 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:50.567689 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:50.567691 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:50.567692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:50.567775 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:50.568064 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:50.568067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:50.568068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:50.568263 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:50.568338 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:50.568574 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 16:38:50.568655 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:38:50.585320 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44319
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:38:50.695226 2487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:38:50.695438 2487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:38:50.695533 2487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:38:50.699357 2487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:38:50.699400 2487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:38:50.699466 2487 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 16:38:50.699497 2487 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 16:38:50.699568 2487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:38:50.704322 2487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:38:50.705255 2487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:38:50.706215 2487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:38:50.715046 2487 server_base.cc:1034] running on GCE node
W20260927 16:38:50.715065 2492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:50.714984 2493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:50.714993 2495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:50.715607 2487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:38:50.716151 2487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:38:50.717324 2487 hybrid_clock.cc:648] HybridClock initialized: now 1790527130717295 us; error 39 us; skew 500 ppm
Sep 27 16:38:50 dist-test-slave-z40b krb5kdc[2090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:38:50.719880 2487 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:38:50.720630 2487 webserver.cc:460] Webserver started at http://127.0.98.3:35995/ using document root <none> and password file <none>
I20260927 16:38:50.721163 2487 fs_manager.cc:359] Metadata directory not provided
I20260927 16:38:50.721243 2487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:38:50.721511 2487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:38:50.723452 2487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-2/data/instance:
uuid: "01d50d647d674fd7a252b717c37cfc49"
format_stamp: "Formatted at 2026-09-27 16:38:50 on dist-test-slave-z40b"
server_key: "78f416090fd01eea2bfd114004a1320e"
server_key_iv: "4eebd85bdf7171670a0f035b32e3abe8"
server_key_version: "encryptionkey@0"
I20260927 16:38:50.724005 2487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-2/wal/instance:
uuid: "01d50d647d674fd7a252b717c37cfc49"
format_stamp: "Formatted at 2026-09-27 16:38:50 on dist-test-slave-z40b"
server_key: "78f416090fd01eea2bfd114004a1320e"
server_key_iv: "4eebd85bdf7171670a0f035b32e3abe8"
server_key_version: "encryptionkey@0"
I20260927 16:38:50.727814 2487 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 16:38:50.730017 2502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:50.731052 2487 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:38:50.731175 2487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "01d50d647d674fd7a252b717c37cfc49"
format_stamp: "Formatted at 2026-09-27 16:38:50 on dist-test-slave-z40b"
server_key: "78f416090fd01eea2bfd114004a1320e"
server_key_iv: "4eebd85bdf7171670a0f035b32e3abe8"
server_key_version: "encryptionkey@0"
I20260927 16:38:50.731309 2487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:38:50.740795 2487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:38:50.743366 2487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:38:50.743541 2487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:38:50.744760 2487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:38:50.744828 2487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:50.744886 2487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:38:50.744935 2487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 16:38:50 dist-test-slave-z40b krb5kdc[2090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:38:50.754987 2487 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36137
I20260927 16:38:50.755807 2487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:38:50.756824 2617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36137 every 8 connection(s)
I20260927 16:38:50.757619 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:50.745500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37101 (local address 127.0.98.62:44319)
0927 16:38:50.745640 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:50.745646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:50.746462 (+ 816us) server_negotiation.cc:407] Connection header received
0927 16:38:50.747543 (+ 1081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:50.747546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:50.747617 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:50.747705 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:50.748726 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.749344 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.750748 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.750994 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.751119 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:50.753855 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:50.753886 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:50.753889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:50.753917 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:50.755597 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:50.756044 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:50.756048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:50.756050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:50.756153 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:50.756533 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:50.756536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:50.756538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:50.756725 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:50.756830 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:50.757331 (+ 501us) server_negotiation.cc:299] Negotiation successful
0927 16:38:50.757452 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 16:38:50.758257 2511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:50.745902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44319 (local address 127.0.98.3:37101)
0927 16:38:50.746319 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:50.746359 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:50.747346 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:50.747876 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:50.747882 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:50.748174 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:50.748584 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:50.748596 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.749465 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.749468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:50.750629 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:50.750638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.751155 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.751159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:50.751353 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:50.751985 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:50.752004 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:50.753697 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:50.755706 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:50.755711 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:50.755722 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:50.755944 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:50.756243 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:50.756248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:50.756250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:50.756419 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:50.756827 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:50.756833 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:50.757101 (+ 268us) client_negotiation.cc:769] Sending connection context
0927 16:38:50.757285 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 16:38:50.757551 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":117,"threads_started":1}
I20260927 16:38:50.759377 2618 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44319
I20260927 16:38:50.759567 2618 heartbeater.cc:464] Registering TS with master...
I20260927 16:38:50.760121 2618 heartbeater.cc:511] Master 127.0.98.62:44319 requested a full tablet report, sending...
I20260927 16:38:50.760222 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 2487
I20260927 16:38:50.760315 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:38:50.760617 392 external_mini_cluster.cc:1428] Setting key 52de3c2325fa34c001d73b6a2e8b1824
I20260927 16:38:50.761152 2135 ts_manager.cc:194] Registered new tserver with Master: 01d50d647d674fd7a252b717c37cfc49 (127.0.98.3:36137)
I20260927 16:38:50.761785 2135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37101
I20260927 16:38:50.762538 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:38:50.769352 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:50.761379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44319 (local address 127.0.98.3:46953)
0927 16:38:50.761764 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:50.761777 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:50.762377 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:50.762637 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:50.762640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:50.762805 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:50.762912 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:50.762918 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.763783 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.763788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:50.764615 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:50.764627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.765263 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.765265 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:50.765332 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:50.765788 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:50.765801 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:50.766087 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:50.767902 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:50.767908 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:50.767913 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:50.768190 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:50.768442 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:50.768447 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:50.768452 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:50.768550 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:50.768998 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:50.769000 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:50.769072 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 16:38:50.769150 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 16:38:50.769240 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":80,"threads_started":1}
I20260927 16:38:50.769531 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:50.761369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46953 (local address 127.0.98.62:44319)
0927 16:38:50.761465 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:50.761468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:50.761854 (+ 386us) server_negotiation.cc:407] Connection header received
0927 16:38:50.762466 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:50.762469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:50.762509 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:50.762578 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:50.763031 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.763664 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.764728 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.765141 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.765255 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:50.766221 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:50.766227 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:50.766234 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:50.766278 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:50.767769 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:50.768287 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:50.768289 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:50.768290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:50.768355 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:50.768656 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:50.768661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:50.768664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:50.768887 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:50.769004 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:50.769231 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 16:38:50.769345 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 16:38:50 dist-test-slave-z40b krb5kdc[2090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 16:38:50 dist-test-slave-z40b krb5kdc[2090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:38:50.786355 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:50.779583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53776 (local address 127.0.98.62:44319)
0927 16:38:50.779681 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:50.779685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:50.779694 (+ 9us) server_negotiation.cc:407] Connection header received
0927 16:38:50.779720 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:50.779721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:50.779749 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:50.779808 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:50.780356 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.780777 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.781459 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.781659 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.781727 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:50.784005 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:50.784007 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:50.784009 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:50.784027 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:50.785069 (+ 1042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:50.785418 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:50.785420 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:50.785421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:50.785485 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:50.785700 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:50.785702 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:50.785703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:50.785839 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:50.785930 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:50.786095 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 16:38:50.786173 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 16:38:50.788815 2134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:38:50.791653 2134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:38:50.804654 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:50.799403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36215 (local address 127.0.0.1:33454)
0927 16:38:50.799981 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:50.799994 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:50.800108 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:50.800572 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:50.800578 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:50.800599 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:38:50.800717 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:50.800724 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.802145 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.802150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:50.803369 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:50.803378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.804306 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.804309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:50.804383 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:50.804407 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 16:38:50.804474 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 16:38:50.804556 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":80,"threads_started":1}
I20260927 16:38:50.804728 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:50.799652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33454 (local address 127.0.98.1:36215)
0927 16:38:50.800077 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:50.800094 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:50.800128 (+ 34us) server_negotiation.cc:407] Connection header received
0927 16:38:50.800211 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:50.800218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:50.800418 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:50.800543 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:38:50.800847 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.802021 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.803482 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.804199 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.804302 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:50.804444 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:50.804499 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 16:38:50.804601 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":95,"threads_started":1}
I20260927 16:38:50.805395 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:50.799704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38761 (local address 127.0.0.1:49228)
0927 16:38:50.800292 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:50.800305 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:50.800379 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:50.801220 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:50.801223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:50.801240 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:38:50.801319 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:50.801328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.802486 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.802492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:50.803825 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:50.803835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.804967 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.804971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:50.805104 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:50.805125 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 16:38:50.805200 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 16:38:50.805279 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":58,"thread_start_us":78,"threads_started":1}
I20260927 16:38:50.805471 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:50.799858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49228 (local address 127.0.98.2:38761)
0927 16:38:50.800767 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:50.800802 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:50.800826 (+ 24us) server_negotiation.cc:407] Connection header received
0927 16:38:50.800898 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:50.800903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:50.801081 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:50.801236 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:38:50.801440 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.802382 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.803950 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.804712 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.804811 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:50.804982 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:50.805239 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 16:38:50.805341 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":811,"thread_start_us":98,"threads_started":1}
I20260927 16:38:50.806393 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:50.799007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36137 (local address 127.0.0.1:37540)
0927 16:38:50.799362 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:50.799407 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:50.799665 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:50.801047 (+ 1382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:50.801050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:50.801066 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:38:50.801151 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:50.801156 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.802651 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.802654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:50.803825 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:50.803835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.806016 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.806020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:50.806117 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:50.806138 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 16:38:50.806216 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 16:38:50.806284 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":51,"thread_start_us":93,"threads_started":1}
I20260927 16:38:50.806461 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:50.799149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37540 (local address 127.0.98.3:36137)
0927 16:38:50.800635 (+ 1486us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:50.800646 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:50.800667 (+ 21us) server_negotiation.cc:407] Connection header received
0927 16:38:50.800733 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:50.800738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:50.800927 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:50.801081 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:38:50.801268 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.802034 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.805056 (+ 3022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:50.805894 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:50.806003 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:50.806156 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:50.806252 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 16:38:50.806336 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1393,"thread_start_us":125,"threads_started":1}
I20260927 16:38:50.807799 2272 tablet_service.cc:1467] Processing CreateTablet for tablet aaf3f4ecf4044a308573c1a6236b2f72 (DEFAULT_TABLE table=test-table [id=4a5e07b155dc4f6d84bb0f0f52fc6e59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:38:50.808061 2412 tablet_service.cc:1467] Processing CreateTablet for tablet aaf3f4ecf4044a308573c1a6236b2f72 (DEFAULT_TABLE table=test-table [id=4a5e07b155dc4f6d84bb0f0f52fc6e59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:38:50.808887 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaf3f4ecf4044a308573c1a6236b2f72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:50.808893 2272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaf3f4ecf4044a308573c1a6236b2f72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:50.809201 2552 tablet_service.cc:1467] Processing CreateTablet for tablet aaf3f4ecf4044a308573c1a6236b2f72 (DEFAULT_TABLE table=test-table [id=4a5e07b155dc4f6d84bb0f0f52fc6e59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:38:50.810042 2552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaf3f4ecf4044a308573c1a6236b2f72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:50.814677 2641 tablet_bootstrap.cc:492] T aaf3f4ecf4044a308573c1a6236b2f72 P 6de1875303d446e1841ca078453646ef: Bootstrap starting.
I20260927 16:38:50.816676 2641 tablet_bootstrap.cc:654] T aaf3f4ecf4044a308573c1a6236b2f72 P 6de1875303d446e1841ca078453646ef: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:50.816711 2643 tablet_bootstrap.cc:492] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49: Bootstrap starting.
I20260927 16:38:50.817219 2642 tablet_bootstrap.cc:492] T aaf3f4ecf4044a308573c1a6236b2f72 P b8c2d915da9f49abbac90af10ecd3337: Bootstrap starting.
I20260927 16:38:50.817540 2641 log.cc:824] T aaf3f4ecf4044a308573c1a6236b2f72 P 6de1875303d446e1841ca078453646ef: Log is configured to *not* fsync() on all Append() calls
I20260927 16:38:50.818571 2643 tablet_bootstrap.cc:654] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:50.819136 2642 tablet_bootstrap.cc:654] T aaf3f4ecf4044a308573c1a6236b2f72 P b8c2d915da9f49abbac90af10ecd3337: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:50.819177 2641 tablet_bootstrap.cc:492] T aaf3f4ecf4044a308573c1a6236b2f72 P 6de1875303d446e1841ca078453646ef: No bootstrap required, opened a new log
I20260927 16:38:50.819398 2641 ts_tablet_manager.cc:1397] T aaf3f4ecf4044a308573c1a6236b2f72 P 6de1875303d446e1841ca078453646ef: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260927 16:38:50.819380 2643 log.cc:824] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49: Log is configured to *not* fsync() on all Append() calls
I20260927 16:38:50.819886 2642 log.cc:824] T aaf3f4ecf4044a308573c1a6236b2f72 P b8c2d915da9f49abbac90af10ecd3337: Log is configured to *not* fsync() on all Append() calls
I20260927 16:38:50.821005 2643 tablet_bootstrap.cc:492] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49: No bootstrap required, opened a new log
I20260927 16:38:50.821172 2643 ts_tablet_manager.cc:1397] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 16:38:50.821553 2642 tablet_bootstrap.cc:492] T aaf3f4ecf4044a308573c1a6236b2f72 P b8c2d915da9f49abbac90af10ecd3337: No bootstrap required, opened a new log
I20260927 16:38:50.821755 2642 ts_tablet_manager.cc:1397] T aaf3f4ecf4044a308573c1a6236b2f72 P b8c2d915da9f49abbac90af10ecd3337: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:38:50.824683 2643 raft_consensus.cc:348] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } }
I20260927 16:38:50.824916 2643 raft_consensus.cc:374] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:50.824966 2643 raft_consensus.cc:729] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d50d647d674fd7a252b717c37cfc49, State: Initialized, Role: FOLLOWER
I20260927 16:38:50.825407 2643 consensus_queue.cc:260] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } }
I20260927 16:38:50.825392 2642 raft_consensus.cc:348] T aaf3f4ecf4044a308573c1a6236b2f72 P b8c2d915da9f49abbac90af10ecd3337 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } }
I20260927 16:38:50.825706 2642 raft_consensus.cc:374] T aaf3f4ecf4044a308573c1a6236b2f72 P b8c2d915da9f49abbac90af10ecd3337 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:50.825759 2642 raft_consensus.cc:729] T aaf3f4ecf4044a308573c1a6236b2f72 P b8c2d915da9f49abbac90af10ecd3337 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8c2d915da9f49abbac90af10ecd3337, State: Initialized, Role: FOLLOWER
I20260927 16:38:50.826169 2642 consensus_queue.cc:260] T aaf3f4ecf4044a308573c1a6236b2f72 P b8c2d915da9f49abbac90af10ecd3337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } }
I20260927 16:38:50.826346 2643 ts_tablet_manager.cc:1428] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 16:38:50.826790 2618 heartbeater.cc:503] Master 127.0.98.62:44319 was elected leader, sending a full tablet report...
I20260927 16:38:50.827023 2338 heartbeater.cc:503] Master 127.0.98.62:44319 was elected leader, sending a full tablet report...
I20260927 16:38:50.828202 2642 ts_tablet_manager.cc:1428] T aaf3f4ecf4044a308573c1a6236b2f72 P b8c2d915da9f49abbac90af10ecd3337: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:38:50.829476 2641 raft_consensus.cc:348] T aaf3f4ecf4044a308573c1a6236b2f72 P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } }
I20260927 16:38:50.829775 2641 raft_consensus.cc:374] T aaf3f4ecf4044a308573c1a6236b2f72 P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:50.829896 2641 raft_consensus.cc:729] T aaf3f4ecf4044a308573c1a6236b2f72 P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6de1875303d446e1841ca078453646ef, State: Initialized, Role: FOLLOWER
I20260927 16:38:50.830572 2641 consensus_queue.cc:260] T aaf3f4ecf4044a308573c1a6236b2f72 P 6de1875303d446e1841ca078453646ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } }
I20260927 16:38:50.831341 2478 heartbeater.cc:503] Master 127.0.98.62:44319 was elected leader, sending a full tablet report...
I20260927 16:38:50.831764 2641 ts_tablet_manager.cc:1428] T aaf3f4ecf4044a308573c1a6236b2f72 P 6de1875303d446e1841ca078453646ef: Time spent starting tablet: real 0.012s user 0.002s sys 0.004s
W20260927 16:38:50.849339 2339 tablet.cc:2369] T aaf3f4ecf4044a308573c1a6236b2f72 P b8c2d915da9f49abbac90af10ecd3337: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:38:51.007434 2619 tablet.cc:2369] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:38:51.057219 2479 tablet.cc:2369] T aaf3f4ecf4044a308573c1a6236b2f72 P 6de1875303d446e1841ca078453646ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:38:51.095471 2647 raft_consensus.cc:482] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:38:51.095633 2647 raft_consensus.cc:504] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } }
I20260927 16:38:51.096702 2647 leader_election.cc:288] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6de1875303d446e1841ca078453646ef (127.0.98.2:38761), b8c2d915da9f49abbac90af10ecd3337 (127.0.98.1:36215)
I20260927 16:38:51.100469 2511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:51.096886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38761 (local address 127.0.98.3:57853)
0927 16:38:51.097058 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:51.097073 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:51.097165 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:51.097435 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:51.097438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:51.097454 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:38:51.097564 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:51.097568 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:51.098499 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:51.098505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:51.099429 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:51.099442 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:51.100087 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:51.100091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:51.100186 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:51.100209 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 16:38:51.100279 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 16:38:51.100342 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 16:38:51.100539 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:51.096945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57853 (local address 127.0.98.2:38761)
0927 16:38:51.097074 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:51.097078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:51.097152 (+ 74us) server_negotiation.cc:407] Connection header received
0927 16:38:51.097294 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:51.097296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:51.097339 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:51.097415 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:38:51.097890 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:51.098389 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:51.099549 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:51.099974 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:51.100073 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:51.100151 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:51.100317 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 16:38:51.100412 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:38:51.101001 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:51.096929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36215 (local address 127.0.98.3:58409)
0927 16:38:51.097280 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:51.097291 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:51.097405 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:51.097658 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:51.097661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:51.097675 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:38:51.098144 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:51.098150 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:51.098856 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:51.098861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:51.099758 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:51.099767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:51.100639 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:51.100644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:51.100738 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:51.100757 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 16:38:51.100822 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 16:38:51.100884 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":95,"threads_started":1}
I20260927 16:38:51.101013 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:51.097067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58409 (local address 127.0.98.1:36215)
0927 16:38:51.097185 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:51.097191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:51.097367 (+ 176us) server_negotiation.cc:407] Connection header received
0927 16:38:51.097520 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:51.097523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:51.097566 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:51.097649 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:38:51.098264 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:51.098748 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:51.099875 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:51.100407 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:51.100498 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:51.100556 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:51.100855 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 16:38:51.100925 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 16:38:51.101120 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaf3f4ecf4044a308573c1a6236b2f72" candidate_uuid: "01d50d647d674fd7a252b717c37cfc49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6de1875303d446e1841ca078453646ef" is_pre_election: true
I20260927 16:38:51.101351 2432 raft_consensus.cc:2454] T aaf3f4ecf4044a308573c1a6236b2f72 P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01d50d647d674fd7a252b717c37cfc49 in term 0.
I20260927 16:38:51.101730 2504 leader_election.cc:302] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d50d647d674fd7a252b717c37cfc49, 6de1875303d446e1841ca078453646ef; no voters:
I20260927 16:38:51.101999 2647 raft_consensus.cc:2781] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:38:51.101994 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaf3f4ecf4044a308573c1a6236b2f72" candidate_uuid: "01d50d647d674fd7a252b717c37cfc49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8c2d915da9f49abbac90af10ecd3337" is_pre_election: true
I20260927 16:38:51.102085 2647 raft_consensus.cc:482] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:38:51.102144 2647 raft_consensus.cc:3037] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:51.102296 2292 raft_consensus.cc:2454] T aaf3f4ecf4044a308573c1a6236b2f72 P b8c2d915da9f49abbac90af10ecd3337 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01d50d647d674fd7a252b717c37cfc49 in term 0.
I20260927 16:38:51.103317 2647 raft_consensus.cc:504] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } }
I20260927 16:38:51.103686 2647 leader_election.cc:288] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49 [CANDIDATE]: Term 1 election: Requested vote from peers 6de1875303d446e1841ca078453646ef (127.0.98.2:38761), b8c2d915da9f49abbac90af10ecd3337 (127.0.98.1:36215)
I20260927 16:38:51.103945 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaf3f4ecf4044a308573c1a6236b2f72" candidate_uuid: "01d50d647d674fd7a252b717c37cfc49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6de1875303d446e1841ca078453646ef"
I20260927 16:38:51.103947 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaf3f4ecf4044a308573c1a6236b2f72" candidate_uuid: "01d50d647d674fd7a252b717c37cfc49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8c2d915da9f49abbac90af10ecd3337"
I20260927 16:38:51.104048 2292 raft_consensus.cc:3037] T aaf3f4ecf4044a308573c1a6236b2f72 P b8c2d915da9f49abbac90af10ecd3337 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:51.104060 2432 raft_consensus.cc:3037] T aaf3f4ecf4044a308573c1a6236b2f72 P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:51.105024 2432 raft_consensus.cc:2454] T aaf3f4ecf4044a308573c1a6236b2f72 P 6de1875303d446e1841ca078453646ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01d50d647d674fd7a252b717c37cfc49 in term 1.
I20260927 16:38:51.105237 2292 raft_consensus.cc:2454] T aaf3f4ecf4044a308573c1a6236b2f72 P b8c2d915da9f49abbac90af10ecd3337 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01d50d647d674fd7a252b717c37cfc49 in term 1.
I20260927 16:38:51.105345 2504 leader_election.cc:302] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d50d647d674fd7a252b717c37cfc49, 6de1875303d446e1841ca078453646ef; no voters:
I20260927 16:38:51.105513 2647 raft_consensus.cc:2781] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:38:51.105697 2647 raft_consensus.cc:686] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49 [term 1 LEADER]: Becoming Leader. State: Replica: 01d50d647d674fd7a252b717c37cfc49, State: Running, Role: LEADER
I20260927 16:38:51.106005 2647 consensus_queue.cc:237] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } }
I20260927 16:38:51.109089 2134 catalog_manager.cc:5495] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01d50d647d674fd7a252b717c37cfc49 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "01d50d647d674fd7a252b717c37cfc49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } health_report { overall_health: UNKNOWN } } }
Sep 27 16:38:51 dist-test-slave-z40b krb5kdc[2090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:38:51.201107 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:51.192925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44319 (local address 127.0.0.1:53786)
0927 16:38:51.193219 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:51.193231 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:51.193354 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:51.193590 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:51.193592 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:51.194096 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:51.194183 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:51.194190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:51.194887 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:51.194893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:51.195514 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:51.195524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:51.195971 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:51.195974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:51.196102 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:51.196596 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:51.196608 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:51.197982 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:51.199543 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:51.199553 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:51.199559 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:51.199791 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:51.200068 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:51.200070 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:51.200072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:51.200167 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:51.200533 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:51.200541 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:51.200651 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 16:38:51.200765 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 16:38:51.200911 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":39,"thread_start_us":101,"threads_started":1}
I20260927 16:38:51.201107 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:51.192965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53786 (local address 127.0.98.62:44319)
0927 16:38:51.193123 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:51.193129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:51.193301 (+ 172us) server_negotiation.cc:407] Connection header received
0927 16:38:51.193449 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:51.193452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:51.193492 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:51.193574 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:51.194295 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:51.194786 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:51.195624 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:51.195860 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:51.195962 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:51.198109 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:51.198120 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:51.198123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:51.198150 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:51.199446 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:51.199883 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:51.199889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:51.199892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:51.199961 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:51.200253 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:51.200259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:51.200261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:51.200418 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:51.200521 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:51.200805 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 16:38:51.200959 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":42}
I20260927 16:38:51.204739 2134 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53786:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 16:38:51.205497 2134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:38:51.210058 2272 tablet_service.cc:1467] Processing CreateTablet for tablet c9d25a75b0214175ab72c41a0698261a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=28dbe2c91ad04849bf20598a2b4ae564]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 16:38:51.210215 2552 tablet_service.cc:1467] Processing CreateTablet for tablet c9d25a75b0214175ab72c41a0698261a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=28dbe2c91ad04849bf20598a2b4ae564]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 16:38:51.210157 2412 tablet_service.cc:1467] Processing CreateTablet for tablet c9d25a75b0214175ab72c41a0698261a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=28dbe2c91ad04849bf20598a2b4ae564]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 16:38:51.210781 2552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d25a75b0214175ab72c41a0698261a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:51.210781 2272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d25a75b0214175ab72c41a0698261a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:51.210840 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d25a75b0214175ab72c41a0698261a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:51.213421 2641 tablet_bootstrap.cc:492] T c9d25a75b0214175ab72c41a0698261a P 6de1875303d446e1841ca078453646ef: Bootstrap starting.
I20260927 16:38:51.213505 2643 tablet_bootstrap.cc:492] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49: Bootstrap starting.
I20260927 16:38:51.214193 2641 tablet_bootstrap.cc:654] T c9d25a75b0214175ab72c41a0698261a P 6de1875303d446e1841ca078453646ef: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:51.214241 2643 tablet_bootstrap.cc:654] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:51.214756 2642 tablet_bootstrap.cc:492] T c9d25a75b0214175ab72c41a0698261a P b8c2d915da9f49abbac90af10ecd3337: Bootstrap starting.
I20260927 16:38:51.215299 2643 tablet_bootstrap.cc:492] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49: No bootstrap required, opened a new log
I20260927 16:38:51.215396 2643 ts_tablet_manager.cc:1397] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 16:38:51.215682 2642 tablet_bootstrap.cc:654] T c9d25a75b0214175ab72c41a0698261a P b8c2d915da9f49abbac90af10ecd3337: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:51.215843 2643 raft_consensus.cc:348] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:51.215926 2643 raft_consensus.cc:374] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:51.215972 2643 raft_consensus.cc:729] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d50d647d674fd7a252b717c37cfc49, State: Initialized, Role: FOLLOWER
I20260927 16:38:51.216089 2643 consensus_queue.cc:260] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:51.216341 2643 ts_tablet_manager.cc:1428] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 16:38:51.216552 2641 tablet_bootstrap.cc:492] T c9d25a75b0214175ab72c41a0698261a P 6de1875303d446e1841ca078453646ef: No bootstrap required, opened a new log
I20260927 16:38:51.216642 2641 ts_tablet_manager.cc:1397] T c9d25a75b0214175ab72c41a0698261a P 6de1875303d446e1841ca078453646ef: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 16:38:51.217098 2641 raft_consensus.cc:348] T c9d25a75b0214175ab72c41a0698261a P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:51.217191 2641 raft_consensus.cc:374] T c9d25a75b0214175ab72c41a0698261a P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:51.217212 2641 raft_consensus.cc:729] T c9d25a75b0214175ab72c41a0698261a P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6de1875303d446e1841ca078453646ef, State: Initialized, Role: FOLLOWER
I20260927 16:38:51.217335 2641 consensus_queue.cc:260] T c9d25a75b0214175ab72c41a0698261a P 6de1875303d446e1841ca078453646ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:51.217602 2641 ts_tablet_manager.cc:1428] T c9d25a75b0214175ab72c41a0698261a P 6de1875303d446e1841ca078453646ef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 16:38:51.217561 2649 tablet_replica.cc:439] T c9d25a75b0214175ab72c41a0698261a P 6de1875303d446e1841ca078453646ef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } } }
I20260927 16:38:51.217679 2649 tablet_replica.cc:442] T c9d25a75b0214175ab72c41a0698261a P 6de1875303d446e1841ca078453646ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:38:51.217794 2647 tablet_replica.cc:439] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } } }
I20260927 16:38:51.217952 2647 tablet_replica.cc:442] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:38:51.218650 2642 tablet_bootstrap.cc:492] T c9d25a75b0214175ab72c41a0698261a P b8c2d915da9f49abbac90af10ecd3337: No bootstrap required, opened a new log
I20260927 16:38:51.218736 2642 ts_tablet_manager.cc:1397] T c9d25a75b0214175ab72c41a0698261a P b8c2d915da9f49abbac90af10ecd3337: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 16:38:51.219137 2642 raft_consensus.cc:348] T c9d25a75b0214175ab72c41a0698261a P b8c2d915da9f49abbac90af10ecd3337 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:51.219302 2642 raft_consensus.cc:374] T c9d25a75b0214175ab72c41a0698261a P b8c2d915da9f49abbac90af10ecd3337 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:51.219345 2642 raft_consensus.cc:729] T c9d25a75b0214175ab72c41a0698261a P b8c2d915da9f49abbac90af10ecd3337 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8c2d915da9f49abbac90af10ecd3337, State: Initialized, Role: FOLLOWER
I20260927 16:38:51.219480 2642 consensus_queue.cc:260] T c9d25a75b0214175ab72c41a0698261a P b8c2d915da9f49abbac90af10ecd3337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:51.219726 2642 ts_tablet_manager.cc:1428] T c9d25a75b0214175ab72c41a0698261a P b8c2d915da9f49abbac90af10ecd3337: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:38:51.219738 2648 tablet_replica.cc:439] T c9d25a75b0214175ab72c41a0698261a P b8c2d915da9f49abbac90af10ecd3337: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } } }
I20260927 16:38:51.219906 2648 tablet_replica.cc:442] T c9d25a75b0214175ab72c41a0698261a P b8c2d915da9f49abbac90af10ecd3337: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:38:51.285590 2647 raft_consensus.cc:482] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:38:51.285763 2647 raft_consensus.cc:504] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:51.286257 2647 leader_election.cc:288] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8c2d915da9f49abbac90af10ecd3337 (127.0.98.1:36215), 6de1875303d446e1841ca078453646ef (127.0.98.2:38761)
I20260927 16:38:51.286824 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d25a75b0214175ab72c41a0698261a" candidate_uuid: "01d50d647d674fd7a252b717c37cfc49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6de1875303d446e1841ca078453646ef" is_pre_election: true
I20260927 16:38:51.286856 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d25a75b0214175ab72c41a0698261a" candidate_uuid: "01d50d647d674fd7a252b717c37cfc49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8c2d915da9f49abbac90af10ecd3337" is_pre_election: true
I20260927 16:38:51.286981 2292 raft_consensus.cc:2454] T c9d25a75b0214175ab72c41a0698261a P b8c2d915da9f49abbac90af10ecd3337 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01d50d647d674fd7a252b717c37cfc49 in term 0.
I20260927 16:38:51.286981 2432 raft_consensus.cc:2454] T c9d25a75b0214175ab72c41a0698261a P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01d50d647d674fd7a252b717c37cfc49 in term 0.
I20260927 16:38:51.287319 2505 leader_election.cc:302] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d50d647d674fd7a252b717c37cfc49, b8c2d915da9f49abbac90af10ecd3337; no voters:
I20260927 16:38:51.287514 2647 raft_consensus.cc:2781] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:38:51.287591 2647 raft_consensus.cc:482] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:38:51.287645 2647 raft_consensus.cc:3037] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:51.288583 2647 raft_consensus.cc:504] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:51.288893 2647 leader_election.cc:288] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49 [CANDIDATE]: Term 1 election: Requested vote from peers b8c2d915da9f49abbac90af10ecd3337 (127.0.98.1:36215), 6de1875303d446e1841ca078453646ef (127.0.98.2:38761)
I20260927 16:38:51.289261 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d25a75b0214175ab72c41a0698261a" candidate_uuid: "01d50d647d674fd7a252b717c37cfc49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8c2d915da9f49abbac90af10ecd3337"
I20260927 16:38:51.289261 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d25a75b0214175ab72c41a0698261a" candidate_uuid: "01d50d647d674fd7a252b717c37cfc49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6de1875303d446e1841ca078453646ef"
I20260927 16:38:51.289384 2292 raft_consensus.cc:3037] T c9d25a75b0214175ab72c41a0698261a P b8c2d915da9f49abbac90af10ecd3337 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:51.289384 2432 raft_consensus.cc:3037] T c9d25a75b0214175ab72c41a0698261a P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:51.290397 2292 raft_consensus.cc:2454] T c9d25a75b0214175ab72c41a0698261a P b8c2d915da9f49abbac90af10ecd3337 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01d50d647d674fd7a252b717c37cfc49 in term 1.
I20260927 16:38:51.290395 2432 raft_consensus.cc:2454] T c9d25a75b0214175ab72c41a0698261a P 6de1875303d446e1841ca078453646ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01d50d647d674fd7a252b717c37cfc49 in term 1.
I20260927 16:38:51.290652 2504 leader_election.cc:302] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d50d647d674fd7a252b717c37cfc49, 6de1875303d446e1841ca078453646ef; no voters:
I20260927 16:38:51.290822 2647 raft_consensus.cc:2781] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:38:51.290915 2647 raft_consensus.cc:686] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49 [term 1 LEADER]: Becoming Leader. State: Replica: 01d50d647d674fd7a252b717c37cfc49, State: Running, Role: LEADER
I20260927 16:38:51.291034 2647 consensus_queue.cc:237] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:51.291733 2651 tablet_replica.cc:439] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49: TxnStatusTablet state changed. Reason: New leader 01d50d647d674fd7a252b717c37cfc49. Latest consensus state: current_term: 1 leader_uuid: "01d50d647d674fd7a252b717c37cfc49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } } }
I20260927 16:38:51.291855 2651 tablet_replica.cc:442] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49: This TxnStatusTablet replica's current role is: LEADER
I20260927 16:38:51.292164 2663 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 16:38:51.292390 2663 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 16:38:51.292694 2135 catalog_manager.cc:5495] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01d50d647d674fd7a252b717c37cfc49 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "01d50d647d674fd7a252b717c37cfc49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } health_report { overall_health: HEALTHY } } }
I20260927 16:38:51.314890 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:51.311066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36137 (local address 127.0.0.1:37544)
0927 16:38:51.311276 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:51.311299 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:51.311381 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:51.311684 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:51.311685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:51.311692 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:38:51.311774 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:51.311778 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:51.312775 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:51.312777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:51.313376 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:51.313384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:51.313933 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:51.313935 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:51.313994 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:51.314031 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:38:51.314677 (+ 646us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:38:51.314686 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 16:38:51.314733 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 16:38:51.314794 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 16:38:51.314930 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:51.311160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37544 (local address 127.0.98.3:36137)
0927 16:38:51.311353 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:51.311358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:51.311372 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:38:51.311504 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:51.311507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:51.311565 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:51.311671 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:38:51.311874 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:51.312662 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:51.313479 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:51.313832 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:51.313931 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:51.314154 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:38:51.314595 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:38:51.314696 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:51.314745 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 16:38:51.314796 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
I20260927 16:38:51.319306 2292 raft_consensus.cc:1261] T c9d25a75b0214175ab72c41a0698261a P b8c2d915da9f49abbac90af10ecd3337 [term 1 FOLLOWER]: Refusing update from remote peer 01d50d647d674fd7a252b717c37cfc49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:38:51.319329 2432 raft_consensus.cc:1261] T c9d25a75b0214175ab72c41a0698261a P 6de1875303d446e1841ca078453646ef [term 1 FOLLOWER]: Refusing update from remote peer 01d50d647d674fd7a252b717c37cfc49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:38:51.319921 2651 consensus_queue.cc:1035] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:38:51.320122 2647 consensus_queue.cc:1035] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:38:51.323860 2648 tablet_replica.cc:439] T c9d25a75b0214175ab72c41a0698261a P b8c2d915da9f49abbac90af10ecd3337: TxnStatusTablet state changed. Reason: New leader 01d50d647d674fd7a252b717c37cfc49. Latest consensus state: current_term: 1 leader_uuid: "01d50d647d674fd7a252b717c37cfc49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } } }
I20260927 16:38:51.323961 2648 tablet_replica.cc:442] T c9d25a75b0214175ab72c41a0698261a P b8c2d915da9f49abbac90af10ecd3337: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:38:51.324679 2649 tablet_replica.cc:439] T c9d25a75b0214175ab72c41a0698261a P 6de1875303d446e1841ca078453646ef: TxnStatusTablet state changed. Reason: New leader 01d50d647d674fd7a252b717c37cfc49. Latest consensus state: current_term: 1 leader_uuid: "01d50d647d674fd7a252b717c37cfc49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } } }
I20260927 16:38:51.324785 2649 tablet_replica.cc:442] T c9d25a75b0214175ab72c41a0698261a P 6de1875303d446e1841ca078453646ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:38:51.327493 2651 tablet_replica.cc:439] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01d50d647d674fd7a252b717c37cfc49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } } }
I20260927 16:38:51.327591 2651 tablet_replica.cc:442] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49: This TxnStatusTablet replica's current role is: LEADER
I20260927 16:38:51.328063 2667 mvcc.cc:205] Tried to move back new op lower bound from 7333999129877688320 to 7333999129769422848. Current Snapshot: MvccSnapshot[applied={T|T < 7333999129877688320}]
I20260927 16:38:51.328246 2648 tablet_replica.cc:439] T c9d25a75b0214175ab72c41a0698261a P b8c2d915da9f49abbac90af10ecd3337: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01d50d647d674fd7a252b717c37cfc49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } } }
I20260927 16:38:51.328334 2648 tablet_replica.cc:442] T c9d25a75b0214175ab72c41a0698261a P b8c2d915da9f49abbac90af10ecd3337: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:38:51.329890 2668 mvcc.cc:205] Tried to move back new op lower bound from 7333999129877688320 to 7333999129769422848. Current Snapshot: MvccSnapshot[applied={T|T < 7333999129877688320}]
I20260927 16:38:51.336789 2647 tablet_replica.cc:439] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01d50d647d674fd7a252b717c37cfc49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } } }
I20260927 16:38:51.336896 2647 tablet_replica.cc:442] T c9d25a75b0214175ab72c41a0698261a P 01d50d647d674fd7a252b717c37cfc49: This TxnStatusTablet replica's current role is: LEADER
I20260927 16:38:51.338099 2669 mvcc.cc:205] Tried to move back new op lower bound from 7333999129877688320 to 7333999129769422848. Current Snapshot: MvccSnapshot[applied={T|T < 7333999129877688320}]
I20260927 16:38:51.338480 2649 tablet_replica.cc:439] T c9d25a75b0214175ab72c41a0698261a P 6de1875303d446e1841ca078453646ef: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01d50d647d674fd7a252b717c37cfc49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } } }
I20260927 16:38:51.338600 2649 tablet_replica.cc:442] T c9d25a75b0214175ab72c41a0698261a P 6de1875303d446e1841ca078453646ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:38:51.340096 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:51.332643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37550 (local address 127.0.98.3:36137)
0927 16:38:51.332772 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:51.332777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:51.335482 (+ 2705us) server_negotiation.cc:407] Connection header received
0927 16:38:51.335639 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:51.335642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:51.335690 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:51.335768 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:38:51.336011 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:51.336709 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:51.338833 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:51.339185 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:51.339301 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:51.339640 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:38:51.339717 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:38:51.339792 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:51.339909 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 16:38:51.339959 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:38:51.350486 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:51.346600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55763 (local address 127.0.98.3:36137)
0927 16:38:51.346773 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:51.346776 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:51.346876 (+ 100us) server_negotiation.cc:407] Connection header received
0927 16:38:51.346996 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:51.346998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:51.347059 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:51.347125 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:38:51.347469 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:51.348137 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:51.349375 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:51.349702 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:51.349796 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:51.350088 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:38:51.350151 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:38:51.350229 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:51.350331 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 16:38:51.350373 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260927 16:38:51.350802 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:51.346495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36137 (local address 127.0.98.3:55763)
0927 16:38:51.346772 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:51.346804 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:51.346902 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:51.347230 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:51.347233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:51.347240 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:38:51.347358 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:51.347364 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:51.348247 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:51.348251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:51.349256 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:51.349265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:51.349809 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:51.349812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:51.349905 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:51.349948 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:38:51.350255 (+ 307us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:38:51.350259 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 16:38:51.350594 (+ 335us) client_negotiation.cc:240] Negotiation successful
0927 16:38:51.350664 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":121,"threads_started":1}
I20260927 16:38:51.357801 2292 raft_consensus.cc:1261] T aaf3f4ecf4044a308573c1a6236b2f72 P b8c2d915da9f49abbac90af10ecd3337 [term 1 FOLLOWER]: Refusing update from remote peer 01d50d647d674fd7a252b717c37cfc49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:38:51.357833 2432 raft_consensus.cc:1261] T aaf3f4ecf4044a308573c1a6236b2f72 P 6de1875303d446e1841ca078453646ef [term 1 FOLLOWER]: Refusing update from remote peer 01d50d647d674fd7a252b717c37cfc49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:38:51.358172 2651 consensus_queue.cc:1035] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:38:51.358515 2651 consensus_queue.cc:1035] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:38:51.393740 2135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:38:51.393903 2135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:38:51.396279 2135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2cc203aec9eb4fe4b7a505cacdf36c3f: Sending DeleteTablet for 3 replicas of tablet aaf3f4ecf4044a308573c1a6236b2f72
I20260927 16:38:51.396857 2552 tablet_service.cc:1514] Processing DeleteTablet for tablet aaf3f4ecf4044a308573c1a6236b2f72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:38:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37540
I20260927 16:38:51.397276 2684 tablet_replica.cc:330] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49: stopping tablet replica
I20260927 16:38:51.397260 2412 tablet_service.cc:1514] Processing DeleteTablet for tablet aaf3f4ecf4044a308573c1a6236b2f72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:38:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49228
I20260927 16:38:51.397689 2684 raft_consensus.cc:2229] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:38:51.397636 2135 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 16:38:51.397742 2272 tablet_service.cc:1514] Processing DeleteTablet for tablet aaf3f4ecf4044a308573c1a6236b2f72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:38:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33454
W20260927 16:38:51.398010 2135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:38:51.398140 2685 tablet_replica.cc:330] T aaf3f4ecf4044a308573c1a6236b2f72 P 6de1875303d446e1841ca078453646ef: stopping tablet replica
I20260927 16:38:51.398371 2684 raft_consensus.cc:2258] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:38:51.398388 2685 raft_consensus.cc:2229] T aaf3f4ecf4044a308573c1a6236b2f72 P 6de1875303d446e1841ca078453646ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:38:51.398552 2685 raft_consensus.cc:2258] T aaf3f4ecf4044a308573c1a6236b2f72 P 6de1875303d446e1841ca078453646ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:38:51.399549 2686 tablet_replica.cc:330] T aaf3f4ecf4044a308573c1a6236b2f72 P b8c2d915da9f49abbac90af10ecd3337: stopping tablet replica
I20260927 16:38:51.399683 2685 ts_tablet_manager.cc:1905] T aaf3f4ecf4044a308573c1a6236b2f72 P 6de1875303d446e1841ca078453646ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:38:51.399794 2686 raft_consensus.cc:2229] T aaf3f4ecf4044a308573c1a6236b2f72 P b8c2d915da9f49abbac90af10ecd3337 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:38:51.399967 2686 raft_consensus.cc:2258] T aaf3f4ecf4044a308573c1a6236b2f72 P b8c2d915da9f49abbac90af10ecd3337 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:38:51.401357 2684 ts_tablet_manager.cc:1905] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:38:51.402576 2685 ts_tablet_manager.cc:1918] T aaf3f4ecf4044a308573c1a6236b2f72 P 6de1875303d446e1841ca078453646ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:38:51.402671 2685 log.cc:1196] T aaf3f4ecf4044a308573c1a6236b2f72 P 6de1875303d446e1841ca078453646ef: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-1/wal/wals/aaf3f4ecf4044a308573c1a6236b2f72
I20260927 16:38:51.403002 2685 ts_tablet_manager.cc:1939] T aaf3f4ecf4044a308573c1a6236b2f72 P 6de1875303d446e1841ca078453646ef: Deleting consensus metadata
I20260927 16:38:51.403375 2272 tablet_service.cc:1467] Processing CreateTablet for tablet c84ffb8068d344d69ed69c31af4f925d (DEFAULT_TABLE table=test-table [id=f916479540b749058f29128195aab035]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:38:51.403635 2272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c84ffb8068d344d69ed69c31af4f925d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:51.403685 2412 tablet_service.cc:1467] Processing CreateTablet for tablet c84ffb8068d344d69ed69c31af4f925d (DEFAULT_TABLE table=test-table [id=f916479540b749058f29128195aab035]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:38:51.403921 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c84ffb8068d344d69ed69c31af4f925d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:51.404578 2684 ts_tablet_manager.cc:1918] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:38:51.404711 2684 log.cc:1196] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-2/wal/wals/aaf3f4ecf4044a308573c1a6236b2f72
I20260927 16:38:51.405082 2684 ts_tablet_manager.cc:1939] T aaf3f4ecf4044a308573c1a6236b2f72 P 01d50d647d674fd7a252b717c37cfc49: Deleting consensus metadata
I20260927 16:38:51.405851 2642 tablet_bootstrap.cc:492] T c84ffb8068d344d69ed69c31af4f925d P b8c2d915da9f49abbac90af10ecd3337: Bootstrap starting.
I20260927 16:38:51.406064 2552 tablet_service.cc:1467] Processing CreateTablet for tablet c84ffb8068d344d69ed69c31af4f925d (DEFAULT_TABLE table=test-table [id=f916479540b749058f29128195aab035]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:38:51.406209 2120 catalog_manager.cc:4841] TS 01d50d647d674fd7a252b717c37cfc49 (127.0.98.3:36137): tablet aaf3f4ecf4044a308573c1a6236b2f72 (table test-table [id=4a5e07b155dc4f6d84bb0f0f52fc6e59]) successfully deleted
I20260927 16:38:51.406307 2552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c84ffb8068d344d69ed69c31af4f925d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:51.406718 2642 tablet_bootstrap.cc:654] T c84ffb8068d344d69ed69c31af4f925d P b8c2d915da9f49abbac90af10ecd3337: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:51.407277 2641 tablet_bootstrap.cc:492] T c84ffb8068d344d69ed69c31af4f925d P 6de1875303d446e1841ca078453646ef: Bootstrap starting.
I20260927 16:38:51.407342 2120 catalog_manager.cc:4841] TS 6de1875303d446e1841ca078453646ef (127.0.98.2:38761): tablet aaf3f4ecf4044a308573c1a6236b2f72 (table test-table [id=4a5e07b155dc4f6d84bb0f0f52fc6e59]) successfully deleted
I20260927 16:38:51.407982 2642 tablet_bootstrap.cc:492] T c84ffb8068d344d69ed69c31af4f925d P b8c2d915da9f49abbac90af10ecd3337: No bootstrap required, opened a new log
I20260927 16:38:51.408077 2642 ts_tablet_manager.cc:1397] T c84ffb8068d344d69ed69c31af4f925d P b8c2d915da9f49abbac90af10ecd3337: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 16:38:51.408121 2643 tablet_bootstrap.cc:492] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49: Bootstrap starting.
I20260927 16:38:51.408223 2641 tablet_bootstrap.cc:654] T c84ffb8068d344d69ed69c31af4f925d P 6de1875303d446e1841ca078453646ef: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:51.408526 2642 raft_consensus.cc:348] T c84ffb8068d344d69ed69c31af4f925d P b8c2d915da9f49abbac90af10ecd3337 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:51.408620 2642 raft_consensus.cc:374] T c84ffb8068d344d69ed69c31af4f925d P b8c2d915da9f49abbac90af10ecd3337 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:51.408658 2642 raft_consensus.cc:729] T c84ffb8068d344d69ed69c31af4f925d P b8c2d915da9f49abbac90af10ecd3337 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8c2d915da9f49abbac90af10ecd3337, State: Initialized, Role: FOLLOWER
I20260927 16:38:51.408772 2642 consensus_queue.cc:260] T c84ffb8068d344d69ed69c31af4f925d P b8c2d915da9f49abbac90af10ecd3337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:51.408978 2643 tablet_bootstrap.cc:654] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:51.409022 2642 ts_tablet_manager.cc:1428] T c84ffb8068d344d69ed69c31af4f925d P b8c2d915da9f49abbac90af10ecd3337: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:38:51.409960 2686 ts_tablet_manager.cc:1905] T aaf3f4ecf4044a308573c1a6236b2f72 P b8c2d915da9f49abbac90af10ecd3337: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:38:51.412475 2686 ts_tablet_manager.cc:1918] T aaf3f4ecf4044a308573c1a6236b2f72 P b8c2d915da9f49abbac90af10ecd3337: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:38:51.412544 2686 log.cc:1196] T aaf3f4ecf4044a308573c1a6236b2f72 P b8c2d915da9f49abbac90af10ecd3337: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-0/wal/wals/aaf3f4ecf4044a308573c1a6236b2f72
I20260927 16:38:51.412803 2686 ts_tablet_manager.cc:1939] T aaf3f4ecf4044a308573c1a6236b2f72 P b8c2d915da9f49abbac90af10ecd3337: Deleting consensus metadata
I20260927 16:38:51.413580 2643 tablet_bootstrap.cc:492] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49: No bootstrap required, opened a new log
I20260927 16:38:51.413673 2643 ts_tablet_manager.cc:1397] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 16:38:51.413930 2121 catalog_manager.cc:4841] TS b8c2d915da9f49abbac90af10ecd3337 (127.0.98.1:36215): tablet aaf3f4ecf4044a308573c1a6236b2f72 (table test-table [id=4a5e07b155dc4f6d84bb0f0f52fc6e59]) successfully deleted
I20260927 16:38:51.414137 2643 raft_consensus.cc:348] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:51.414287 2643 raft_consensus.cc:374] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:51.414314 2643 raft_consensus.cc:729] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d50d647d674fd7a252b717c37cfc49, State: Initialized, Role: FOLLOWER
I20260927 16:38:51.414433 2643 consensus_queue.cc:260] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:51.414700 2643 ts_tablet_manager.cc:1428] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 16:38:51.415900 2641 tablet_bootstrap.cc:492] T c84ffb8068d344d69ed69c31af4f925d P 6de1875303d446e1841ca078453646ef: No bootstrap required, opened a new log
I20260927 16:38:51.415984 2641 ts_tablet_manager.cc:1397] T c84ffb8068d344d69ed69c31af4f925d P 6de1875303d446e1841ca078453646ef: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260927 16:38:51.416399 2641 raft_consensus.cc:348] T c84ffb8068d344d69ed69c31af4f925d P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:51.416509 2641 raft_consensus.cc:374] T c84ffb8068d344d69ed69c31af4f925d P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:51.416541 2641 raft_consensus.cc:729] T c84ffb8068d344d69ed69c31af4f925d P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6de1875303d446e1841ca078453646ef, State: Initialized, Role: FOLLOWER
I20260927 16:38:51.416656 2641 consensus_queue.cc:260] T c84ffb8068d344d69ed69c31af4f925d P 6de1875303d446e1841ca078453646ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:51.416903 2641 ts_tablet_manager.cc:1428] T c84ffb8068d344d69ed69c31af4f925d P 6de1875303d446e1841ca078453646ef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:38:51.451848 2676 raft_consensus.cc:482] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:38:51.451947 2676 raft_consensus.cc:504] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:51.452375 2676 leader_election.cc:288] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8c2d915da9f49abbac90af10ecd3337 (127.0.98.1:36215), 6de1875303d446e1841ca078453646ef (127.0.98.2:38761)
I20260927 16:38:51.452702 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c84ffb8068d344d69ed69c31af4f925d" candidate_uuid: "01d50d647d674fd7a252b717c37cfc49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8c2d915da9f49abbac90af10ecd3337" is_pre_election: true
I20260927 16:38:51.452745 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c84ffb8068d344d69ed69c31af4f925d" candidate_uuid: "01d50d647d674fd7a252b717c37cfc49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6de1875303d446e1841ca078453646ef" is_pre_election: true
I20260927 16:38:51.452837 2291 raft_consensus.cc:2454] T c84ffb8068d344d69ed69c31af4f925d P b8c2d915da9f49abbac90af10ecd3337 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01d50d647d674fd7a252b717c37cfc49 in term 0.
I20260927 16:38:51.452848 2432 raft_consensus.cc:2454] T c84ffb8068d344d69ed69c31af4f925d P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01d50d647d674fd7a252b717c37cfc49 in term 0.
I20260927 16:38:51.453073 2504 leader_election.cc:302] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d50d647d674fd7a252b717c37cfc49, 6de1875303d446e1841ca078453646ef; no voters:
I20260927 16:38:51.453250 2676 raft_consensus.cc:2781] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:38:51.453290 2676 raft_consensus.cc:482] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:38:51.453331 2676 raft_consensus.cc:3037] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:51.454145 2676 raft_consensus.cc:504] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:51.454453 2676 leader_election.cc:288] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49 [CANDIDATE]: Term 1 election: Requested vote from peers b8c2d915da9f49abbac90af10ecd3337 (127.0.98.1:36215), 6de1875303d446e1841ca078453646ef (127.0.98.2:38761)
I20260927 16:38:51.454710 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c84ffb8068d344d69ed69c31af4f925d" candidate_uuid: "01d50d647d674fd7a252b717c37cfc49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6de1875303d446e1841ca078453646ef"
I20260927 16:38:51.454700 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c84ffb8068d344d69ed69c31af4f925d" candidate_uuid: "01d50d647d674fd7a252b717c37cfc49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8c2d915da9f49abbac90af10ecd3337"
I20260927 16:38:51.454819 2432 raft_consensus.cc:3037] T c84ffb8068d344d69ed69c31af4f925d P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:51.454813 2291 raft_consensus.cc:3037] T c84ffb8068d344d69ed69c31af4f925d P b8c2d915da9f49abbac90af10ecd3337 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:51.455700 2432 raft_consensus.cc:2454] T c84ffb8068d344d69ed69c31af4f925d P 6de1875303d446e1841ca078453646ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01d50d647d674fd7a252b717c37cfc49 in term 1.
I20260927 16:38:51.455865 2291 raft_consensus.cc:2454] T c84ffb8068d344d69ed69c31af4f925d P b8c2d915da9f49abbac90af10ecd3337 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01d50d647d674fd7a252b717c37cfc49 in term 1.
I20260927 16:38:51.455989 2504 leader_election.cc:302] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d50d647d674fd7a252b717c37cfc49, 6de1875303d446e1841ca078453646ef; no voters:
I20260927 16:38:51.456198 2676 raft_consensus.cc:2781] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:38:51.456280 2676 raft_consensus.cc:686] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49 [term 1 LEADER]: Becoming Leader. State: Replica: 01d50d647d674fd7a252b717c37cfc49, State: Running, Role: LEADER
I20260927 16:38:51.456396 2676 consensus_queue.cc:237] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:51.457805 2135 catalog_manager.cc:5495] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01d50d647d674fd7a252b717c37cfc49 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "01d50d647d674fd7a252b717c37cfc49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } health_report { overall_health: HEALTHY } } }
I20260927 16:38:51.480921 2291 raft_consensus.cc:1261] T c84ffb8068d344d69ed69c31af4f925d P b8c2d915da9f49abbac90af10ecd3337 [term 1 FOLLOWER]: Refusing update from remote peer 01d50d647d674fd7a252b717c37cfc49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:38:51.481128 2431 raft_consensus.cc:1261] T c84ffb8068d344d69ed69c31af4f925d P 6de1875303d446e1841ca078453646ef [term 1 FOLLOWER]: Refusing update from remote peer 01d50d647d674fd7a252b717c37cfc49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:38:51.481343 2676 consensus_queue.cc:1035] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:38:51.481612 2647 consensus_queue.cc:1035] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 16:38:51.494490 2667 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 16:38:53.041960 2134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:38:53.042096 2134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:38:53.044183 2134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2cc203aec9eb4fe4b7a505cacdf36c3f: Sending DeleteTablet for 3 replicas of tablet c84ffb8068d344d69ed69c31af4f925d
I20260927 16:38:53.044685 2412 tablet_service.cc:1514] Processing DeleteTablet for tablet c84ffb8068d344d69ed69c31af4f925d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:38:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49228
I20260927 16:38:53.044776 2552 tablet_service.cc:1514] Processing DeleteTablet for tablet c84ffb8068d344d69ed69c31af4f925d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:38:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37540
I20260927 16:38:53.044802 2272 tablet_service.cc:1514] Processing DeleteTablet for tablet c84ffb8068d344d69ed69c31af4f925d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:38:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33454
I20260927 16:38:53.045040 2714 tablet_replica.cc:330] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49: stopping tablet replica
I20260927 16:38:53.045147 2715 tablet_replica.cc:330] T c84ffb8068d344d69ed69c31af4f925d P b8c2d915da9f49abbac90af10ecd3337: stopping tablet replica
I20260927 16:38:53.045224 2714 raft_consensus.cc:2229] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:38:53.045389 2715 raft_consensus.cc:2229] T c84ffb8068d344d69ed69c31af4f925d P b8c2d915da9f49abbac90af10ecd3337 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:38:53.045514 2715 raft_consensus.cc:2258] T c84ffb8068d344d69ed69c31af4f925d P b8c2d915da9f49abbac90af10ecd3337 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:38:53.046132 2715 ts_tablet_manager.cc:1905] T c84ffb8068d344d69ed69c31af4f925d P b8c2d915da9f49abbac90af10ecd3337: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:38:53.047892 2714 raft_consensus.cc:2258] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:38:53.048266 2713 tablet_replica.cc:330] T c84ffb8068d344d69ed69c31af4f925d P 6de1875303d446e1841ca078453646ef: stopping tablet replica
I20260927 16:38:53.048384 2713 raft_consensus.cc:2229] T c84ffb8068d344d69ed69c31af4f925d P 6de1875303d446e1841ca078453646ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:38:53.048488 2714 ts_tablet_manager.cc:1905] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:38:53.048501 2713 raft_consensus.cc:2258] T c84ffb8068d344d69ed69c31af4f925d P 6de1875303d446e1841ca078453646ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:38:53.049023 2713 ts_tablet_manager.cc:1905] T c84ffb8068d344d69ed69c31af4f925d P 6de1875303d446e1841ca078453646ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:38:53.049022 2715 ts_tablet_manager.cc:1918] T c84ffb8068d344d69ed69c31af4f925d P b8c2d915da9f49abbac90af10ecd3337: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 16:38:53.049100 2715 log.cc:1196] T c84ffb8068d344d69ed69c31af4f925d P b8c2d915da9f49abbac90af10ecd3337: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-0/wal/wals/c84ffb8068d344d69ed69c31af4f925d
I20260927 16:38:53.049372 2715 ts_tablet_manager.cc:1939] T c84ffb8068d344d69ed69c31af4f925d P b8c2d915da9f49abbac90af10ecd3337: Deleting consensus metadata
I20260927 16:38:53.049947 2121 catalog_manager.cc:4841] TS b8c2d915da9f49abbac90af10ecd3337 (127.0.98.1:36215): tablet c84ffb8068d344d69ed69c31af4f925d (table test-table [id=f916479540b749058f29128195aab035]) successfully deleted
I20260927 16:38:53.051252 2713 ts_tablet_manager.cc:1918] T c84ffb8068d344d69ed69c31af4f925d P 6de1875303d446e1841ca078453646ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 16:38:53.051340 2713 log.cc:1196] T c84ffb8068d344d69ed69c31af4f925d P 6de1875303d446e1841ca078453646ef: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-1/wal/wals/c84ffb8068d344d69ed69c31af4f925d
I20260927 16:38:53.051604 2713 ts_tablet_manager.cc:1939] T c84ffb8068d344d69ed69c31af4f925d P 6de1875303d446e1841ca078453646ef: Deleting consensus metadata
I20260927 16:38:53.052174 2120 catalog_manager.cc:4841] TS 6de1875303d446e1841ca078453646ef (127.0.98.2:38761): tablet c84ffb8068d344d69ed69c31af4f925d (table test-table [id=f916479540b749058f29128195aab035]) successfully deleted
I20260927 16:38:53.055078 2714 ts_tablet_manager.cc:1918] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 16:38:53.055166 2714 log.cc:1196] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-2/wal/wals/c84ffb8068d344d69ed69c31af4f925d
I20260927 16:38:53.055419 2714 ts_tablet_manager.cc:1939] T c84ffb8068d344d69ed69c31af4f925d P 01d50d647d674fd7a252b717c37cfc49: Deleting consensus metadata
I20260927 16:38:53.055927 2120 catalog_manager.cc:4841] TS 01d50d647d674fd7a252b717c37cfc49 (127.0.98.3:36137): tablet c84ffb8068d344d69ed69c31af4f925d (table test-table [id=f916479540b749058f29128195aab035]) successfully deleted
Sep 27 16:38:53 dist-test-slave-z40b krb5kdc[2090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527133, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 16:38:53 dist-test-slave-z40b krb5kdc[2090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527133, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:38:53.074162 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:53.062994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53798 (local address 127.0.98.62:44319)
0927 16:38:53.063256 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:53.063263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:53.063479 (+ 216us) server_negotiation.cc:407] Connection header received
0927 16:38:53.063578 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:53.063580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:53.063620 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:53.063699 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:53.064431 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:53.064929 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:53.065755 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:53.065952 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:53.066020 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:53.071512 (+ 5492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:53.071515 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:53.071517 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:53.071536 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:53.072959 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:53.073288 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:53.073290 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:53.073291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:53.073359 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:53.073556 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:53.073558 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:53.073559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:53.073717 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:53.073809 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:53.073962 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 16:38:53.074047 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":76,"threads_started":1}
I20260927 16:38:53.076272 2134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:38:53.076736 2134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:38:53.080854 2272 tablet_service.cc:1467] Processing CreateTablet for tablet ad907f4844fa4797aa7fc4f74e0678c6 (DEFAULT_TABLE table=test-table [id=78ac7410e2d04ac69817b48e9d033fd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:38:53.080897 2552 tablet_service.cc:1467] Processing CreateTablet for tablet ad907f4844fa4797aa7fc4f74e0678c6 (DEFAULT_TABLE table=test-table [id=78ac7410e2d04ac69817b48e9d033fd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:38:53.081126 2552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad907f4844fa4797aa7fc4f74e0678c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:53.081127 2272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad907f4844fa4797aa7fc4f74e0678c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:53.083364 2724 tablet_bootstrap.cc:492] T ad907f4844fa4797aa7fc4f74e0678c6 P b8c2d915da9f49abbac90af10ecd3337: Bootstrap starting.
I20260927 16:38:53.083525 2412 tablet_service.cc:1467] Processing CreateTablet for tablet ad907f4844fa4797aa7fc4f74e0678c6 (DEFAULT_TABLE table=test-table [id=78ac7410e2d04ac69817b48e9d033fd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:38:53.083791 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad907f4844fa4797aa7fc4f74e0678c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:53.084548 2724 tablet_bootstrap.cc:654] T ad907f4844fa4797aa7fc4f74e0678c6 P b8c2d915da9f49abbac90af10ecd3337: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:53.085136 2725 tablet_bootstrap.cc:492] T ad907f4844fa4797aa7fc4f74e0678c6 P 01d50d647d674fd7a252b717c37cfc49: Bootstrap starting.
I20260927 16:38:53.085690 2724 tablet_bootstrap.cc:492] T ad907f4844fa4797aa7fc4f74e0678c6 P b8c2d915da9f49abbac90af10ecd3337: No bootstrap required, opened a new log
I20260927 16:38:53.085740 2727 tablet_bootstrap.cc:492] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef: Bootstrap starting.
I20260927 16:38:53.085860 2724 ts_tablet_manager.cc:1397] T ad907f4844fa4797aa7fc4f74e0678c6 P b8c2d915da9f49abbac90af10ecd3337: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 16:38:53.085997 2725 tablet_bootstrap.cc:654] T ad907f4844fa4797aa7fc4f74e0678c6 P 01d50d647d674fd7a252b717c37cfc49: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:53.086666 2727 tablet_bootstrap.cc:654] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:53.086755 2724 raft_consensus.cc:348] T ad907f4844fa4797aa7fc4f74e0678c6 P b8c2d915da9f49abbac90af10ecd3337 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:53.086874 2724 raft_consensus.cc:374] T ad907f4844fa4797aa7fc4f74e0678c6 P b8c2d915da9f49abbac90af10ecd3337 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:53.086913 2724 raft_consensus.cc:729] T ad907f4844fa4797aa7fc4f74e0678c6 P b8c2d915da9f49abbac90af10ecd3337 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8c2d915da9f49abbac90af10ecd3337, State: Initialized, Role: FOLLOWER
I20260927 16:38:53.087049 2724 consensus_queue.cc:260] T ad907f4844fa4797aa7fc4f74e0678c6 P b8c2d915da9f49abbac90af10ecd3337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:53.087177 2725 tablet_bootstrap.cc:492] T ad907f4844fa4797aa7fc4f74e0678c6 P 01d50d647d674fd7a252b717c37cfc49: No bootstrap required, opened a new log
I20260927 16:38:53.087291 2725 ts_tablet_manager.cc:1397] T ad907f4844fa4797aa7fc4f74e0678c6 P 01d50d647d674fd7a252b717c37cfc49: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 16:38:53.087488 2724 ts_tablet_manager.cc:1428] T ad907f4844fa4797aa7fc4f74e0678c6 P b8c2d915da9f49abbac90af10ecd3337: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 16:38:53.087692 2725 raft_consensus.cc:348] T ad907f4844fa4797aa7fc4f74e0678c6 P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:53.087766 2725 raft_consensus.cc:374] T ad907f4844fa4797aa7fc4f74e0678c6 P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:53.087829 2725 raft_consensus.cc:729] T ad907f4844fa4797aa7fc4f74e0678c6 P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d50d647d674fd7a252b717c37cfc49, State: Initialized, Role: FOLLOWER
I20260927 16:38:53.087859 2727 tablet_bootstrap.cc:492] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef: No bootstrap required, opened a new log
I20260927 16:38:53.087941 2727 ts_tablet_manager.cc:1397] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 16:38:53.087989 2725 consensus_queue.cc:260] T ad907f4844fa4797aa7fc4f74e0678c6 P 01d50d647d674fd7a252b717c37cfc49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:53.088253 2725 ts_tablet_manager.cc:1428] T ad907f4844fa4797aa7fc4f74e0678c6 P 01d50d647d674fd7a252b717c37cfc49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:38:53.088383 2727 raft_consensus.cc:348] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:53.088495 2727 raft_consensus.cc:374] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:53.088531 2727 raft_consensus.cc:729] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6de1875303d446e1841ca078453646ef, State: Initialized, Role: FOLLOWER
I20260927 16:38:53.088671 2727 consensus_queue.cc:260] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:53.089017 2727 ts_tablet_manager.cc:1428] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:38:53.434893 2731 raft_consensus.cc:482] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:38:53.435047 2731 raft_consensus.cc:504] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:53.436247 2731 leader_election.cc:288] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8c2d915da9f49abbac90af10ecd3337 (127.0.98.1:36215), 01d50d647d674fd7a252b717c37cfc49 (127.0.98.3:36137)
I20260927 16:38:53.440548 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:53.436577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52927 (local address 127.0.98.1:36215)
0927 16:38:53.436861 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:53.436871 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:53.436990 (+ 119us) server_negotiation.cc:407] Connection header received
0927 16:38:53.437170 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:53.437172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:53.437212 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:53.437300 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:38:53.437590 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:53.438109 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:53.439443 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:53.439998 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:53.440097 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:53.440177 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:53.440313 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 16:38:53.440387 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":110,"threads_started":1}
I20260927 16:38:53.440557 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:53.436474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36215 (local address 127.0.98.2:52927)
0927 16:38:53.436873 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:53.436887 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:53.437039 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:53.437323 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:53.437326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:53.437342 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:38:53.437466 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:53.437471 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:53.438248 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:53.438251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:53.439336 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:53.439346 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:53.440105 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:53.440109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:53.440205 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:53.440229 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 16:38:53.440300 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 16:38:53.440363 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":26,"spinlock_wait_cycles":4224,"thread_start_us":117,"threads_started":1}
I20260927 16:38:53.440938 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad907f4844fa4797aa7fc4f74e0678c6" candidate_uuid: "6de1875303d446e1841ca078453646ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8c2d915da9f49abbac90af10ecd3337" is_pre_election: true
I20260927 16:38:53.441009 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:53.436470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36137 (local address 127.0.98.2:44937)
0927 16:38:53.436860 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:53.436875 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:53.437041 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:53.437296 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:53.437299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:53.437314 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:38:53.437435 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:53.437440 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:53.438469 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:53.438472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:53.439611 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:53.439620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:53.440566 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:53.440572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:53.440697 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:53.440731 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 16:38:53.440804 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 16:38:53.440870 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":70,"spinlock_wait_cycles":36480,"thread_start_us":116,"threads_started":1}
I20260927 16:38:53.441025 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:53.436540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44937 (local address 127.0.98.3:36137)
0927 16:38:53.436840 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:53.436848 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:53.436967 (+ 119us) server_negotiation.cc:407] Connection header received
0927 16:38:53.437146 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:53.437149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:53.437200 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:53.437266 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:38:53.437535 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:53.438092 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:53.439750 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:53.440357 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:53.440456 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:53.440510 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:53.440831 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 16:38:53.440903 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":108,"threads_started":1}
I20260927 16:38:53.441087 2291 raft_consensus.cc:2454] T ad907f4844fa4797aa7fc4f74e0678c6 P b8c2d915da9f49abbac90af10ecd3337 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6de1875303d446e1841ca078453646ef in term 0.
I20260927 16:38:53.441468 2365 leader_election.cc:302] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6de1875303d446e1841ca078453646ef, b8c2d915da9f49abbac90af10ecd3337; no voters:
I20260927 16:38:53.441541 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad907f4844fa4797aa7fc4f74e0678c6" candidate_uuid: "6de1875303d446e1841ca078453646ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d50d647d674fd7a252b717c37cfc49" is_pre_election: true
I20260927 16:38:53.441730 2731 raft_consensus.cc:2781] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:38:53.441785 2731 raft_consensus.cc:482] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:38:53.441761 2572 raft_consensus.cc:2454] T ad907f4844fa4797aa7fc4f74e0678c6 P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6de1875303d446e1841ca078453646ef in term 0.
I20260927 16:38:53.441824 2731 raft_consensus.cc:3037] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:53.442870 2731 raft_consensus.cc:504] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:53.443150 2731 leader_election.cc:288] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef [CANDIDATE]: Term 1 election: Requested vote from peers b8c2d915da9f49abbac90af10ecd3337 (127.0.98.1:36215), 01d50d647d674fd7a252b717c37cfc49 (127.0.98.3:36137)
I20260927 16:38:53.443440 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad907f4844fa4797aa7fc4f74e0678c6" candidate_uuid: "6de1875303d446e1841ca078453646ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8c2d915da9f49abbac90af10ecd3337"
I20260927 16:38:53.443486 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad907f4844fa4797aa7fc4f74e0678c6" candidate_uuid: "6de1875303d446e1841ca078453646ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d50d647d674fd7a252b717c37cfc49"
I20260927 16:38:53.443537 2291 raft_consensus.cc:3037] T ad907f4844fa4797aa7fc4f74e0678c6 P b8c2d915da9f49abbac90af10ecd3337 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:53.443611 2572 raft_consensus.cc:3037] T ad907f4844fa4797aa7fc4f74e0678c6 P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:53.444406 2291 raft_consensus.cc:2454] T ad907f4844fa4797aa7fc4f74e0678c6 P b8c2d915da9f49abbac90af10ecd3337 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6de1875303d446e1841ca078453646ef in term 1.
I20260927 16:38:53.444550 2572 raft_consensus.cc:2454] T ad907f4844fa4797aa7fc4f74e0678c6 P 01d50d647d674fd7a252b717c37cfc49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6de1875303d446e1841ca078453646ef in term 1.
I20260927 16:38:53.444674 2365 leader_election.cc:302] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6de1875303d446e1841ca078453646ef, b8c2d915da9f49abbac90af10ecd3337; no voters:
I20260927 16:38:53.444842 2731 raft_consensus.cc:2781] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:38:53.445055 2731 raft_consensus.cc:686] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef [term 1 LEADER]: Becoming Leader. State: Replica: 6de1875303d446e1841ca078453646ef, State: Running, Role: LEADER
I20260927 16:38:53.445333 2731 consensus_queue.cc:237] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:53.447377 2134 catalog_manager.cc:5495] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 6de1875303d446e1841ca078453646ef (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6de1875303d446e1841ca078453646ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } health_report { overall_health: UNKNOWN } } }
I20260927 16:38:53.538995 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:53.535284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49244 (local address 127.0.98.2:38761)
0927 16:38:53.535564 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:53.535575 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:53.535587 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:38:53.535666 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:53.535668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:53.535715 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:53.535837 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:38:53.536079 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:53.536629 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:53.537550 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:53.537782 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:53.537883 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:53.538180 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:38:53.538621 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:38:53.538752 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:53.538791 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 16:38:53.538840 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":107,"threads_started":1}
I20260927 16:38:53.547971 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:53.544854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39651 (local address 127.0.98.3:36137)
0927 16:38:53.544959 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:53.544962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:53.545102 (+ 140us) server_negotiation.cc:407] Connection header received
0927 16:38:53.545224 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:53.545226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:53.545266 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:53.545325 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:38:53.545912 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:53.546381 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:53.547110 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:53.547349 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:53.547419 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:53.547641 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:38:53.547707 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:38:53.547756 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:53.547843 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 16:38:53.547880 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 16:38:53.547983 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:53.544765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36137 (local address 127.0.98.2:39651)
0927 16:38:53.545021 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:53.545033 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:53.545141 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:53.545346 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:53.545348 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:53.545354 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:38:53.545823 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:53.545828 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:53.546462 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:53.546464 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:53.547023 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:53.547030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:53.547432 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:53.547435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:53.547504 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:53.547541 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:38:53.547772 (+ 231us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:38:53.547776 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 16:38:53.547823 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 16:38:53.547877 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":101,"threads_started":1}
I20260927 16:38:53.553331 2291 raft_consensus.cc:1261] T ad907f4844fa4797aa7fc4f74e0678c6 P b8c2d915da9f49abbac90af10ecd3337 [term 1 FOLLOWER]: Refusing update from remote peer 6de1875303d446e1841ca078453646ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:38:53.553491 2572 raft_consensus.cc:1261] T ad907f4844fa4797aa7fc4f74e0678c6 P 01d50d647d674fd7a252b717c37cfc49 [term 1 FOLLOWER]: Refusing update from remote peer 6de1875303d446e1841ca078453646ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:38:53.553730 2731 consensus_queue.cc:1035] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:38:53.553903 2736 consensus_queue.cc:1035] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:38:53.571810 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:53.568519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60875 (local address 127.0.98.2:38761)
0927 16:38:53.568614 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:53.568618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:53.568785 (+ 167us) server_negotiation.cc:407] Connection header received
0927 16:38:53.568959 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:53.568962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:53.569009 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:53.569085 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:38:53.569418 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:53.569919 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:53.570707 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:53.571012 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:53.571130 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:53.571383 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:38:53.571460 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:38:53.571519 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:53.571635 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 16:38:53.571683 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 16:38:53.572170 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:53.568475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38761 (local address 127.0.98.3:60875)
0927 16:38:53.568713 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:53.568727 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:53.568841 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:53.569191 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:53.569195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:53.569205 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:38:53.569317 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:53.569322 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:53.570020 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:53.570026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:53.570598 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:53.570606 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:53.571142 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:53.571145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:53.571247 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:53.571276 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:38:53.571537 (+ 261us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:38:53.571544 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 16:38:53.571923 (+ 379us) client_negotiation.cc:240] Negotiation successful
0927 16:38:53.572010 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":94,"threads_started":1}
I20260927 16:38:53.592625 2133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:38:53.592775 2133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:38:53.594638 2133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2cc203aec9eb4fe4b7a505cacdf36c3f: Sending DeleteTablet for 3 replicas of tablet ad907f4844fa4797aa7fc4f74e0678c6
I20260927 16:38:53.595108 2272 tablet_service.cc:1514] Processing DeleteTablet for tablet ad907f4844fa4797aa7fc4f74e0678c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:38:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33454
I20260927 16:38:53.595196 2412 tablet_service.cc:1514] Processing DeleteTablet for tablet ad907f4844fa4797aa7fc4f74e0678c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:38:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49228
I20260927 16:38:53.595247 2552 tablet_service.cc:1514] Processing DeleteTablet for tablet ad907f4844fa4797aa7fc4f74e0678c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:38:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37540
I20260927 16:38:53.595505 2752 tablet_replica.cc:330] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef: stopping tablet replica
I20260927 16:38:53.595628 2752 raft_consensus.cc:2229] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:38:53.595724 2751 tablet_replica.cc:330] T ad907f4844fa4797aa7fc4f74e0678c6 P b8c2d915da9f49abbac90af10ecd3337: stopping tablet replica
I20260927 16:38:53.595875 2751 raft_consensus.cc:2229] T ad907f4844fa4797aa7fc4f74e0678c6 P b8c2d915da9f49abbac90af10ecd3337 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:38:53.595882 2752 raft_consensus.cc:2258] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:38:53.596033 2751 raft_consensus.cc:2258] T ad907f4844fa4797aa7fc4f74e0678c6 P b8c2d915da9f49abbac90af10ecd3337 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:38:53.596101 2753 tablet_replica.cc:330] T ad907f4844fa4797aa7fc4f74e0678c6 P 01d50d647d674fd7a252b717c37cfc49: stopping tablet replica
I20260927 16:38:53.596235 2753 raft_consensus.cc:2229] T ad907f4844fa4797aa7fc4f74e0678c6 P 01d50d647d674fd7a252b717c37cfc49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:38:53.596180 2133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 16:38:53.596333 2753 raft_consensus.cc:2258] T ad907f4844fa4797aa7fc4f74e0678c6 P 01d50d647d674fd7a252b717c37cfc49 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 16:38:53.596545 2133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:38:53.596915 2751 ts_tablet_manager.cc:1905] T ad907f4844fa4797aa7fc4f74e0678c6 P b8c2d915da9f49abbac90af10ecd3337: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:38:53.596930 2753 ts_tablet_manager.cc:1905] T ad907f4844fa4797aa7fc4f74e0678c6 P 01d50d647d674fd7a252b717c37cfc49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:38:53.599603 2753 ts_tablet_manager.cc:1918] T ad907f4844fa4797aa7fc4f74e0678c6 P 01d50d647d674fd7a252b717c37cfc49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:38:53.599687 2753 log.cc:1196] T ad907f4844fa4797aa7fc4f74e0678c6 P 01d50d647d674fd7a252b717c37cfc49: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-2/wal/wals/ad907f4844fa4797aa7fc4f74e0678c6
I20260927 16:38:53.599771 2751 ts_tablet_manager.cc:1918] T ad907f4844fa4797aa7fc4f74e0678c6 P b8c2d915da9f49abbac90af10ecd3337: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:38:53.599895 2751 log.cc:1196] T ad907f4844fa4797aa7fc4f74e0678c6 P b8c2d915da9f49abbac90af10ecd3337: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-0/wal/wals/ad907f4844fa4797aa7fc4f74e0678c6
I20260927 16:38:53.599956 2753 ts_tablet_manager.cc:1939] T ad907f4844fa4797aa7fc4f74e0678c6 P 01d50d647d674fd7a252b717c37cfc49: Deleting consensus metadata
I20260927 16:38:53.600302 2751 ts_tablet_manager.cc:1939] T ad907f4844fa4797aa7fc4f74e0678c6 P b8c2d915da9f49abbac90af10ecd3337: Deleting consensus metadata
I20260927 16:38:53.600608 2120 catalog_manager.cc:4841] TS 01d50d647d674fd7a252b717c37cfc49 (127.0.98.3:36137): tablet ad907f4844fa4797aa7fc4f74e0678c6 (table test-table [id=78ac7410e2d04ac69817b48e9d033fd5]) successfully deleted
I20260927 16:38:53.600960 2752 ts_tablet_manager.cc:1905] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:38:53.601050 2121 catalog_manager.cc:4841] TS b8c2d915da9f49abbac90af10ecd3337 (127.0.98.1:36215): tablet ad907f4844fa4797aa7fc4f74e0678c6 (table test-table [id=78ac7410e2d04ac69817b48e9d033fd5]) successfully deleted
I20260927 16:38:53.601205 2412 tablet_service.cc:1467] Processing CreateTablet for tablet 1183ef1121ed458b855a2f44dc6d15ad (DEFAULT_TABLE table=test-table [id=7e6b9a4fd86c40a5a0b6c1deae5fb4f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:38:53.601418 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1183ef1121ed458b855a2f44dc6d15ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:53.601507 2552 tablet_service.cc:1467] Processing CreateTablet for tablet 1183ef1121ed458b855a2f44dc6d15ad (DEFAULT_TABLE table=test-table [id=7e6b9a4fd86c40a5a0b6c1deae5fb4f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:38:53.601737 2552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1183ef1121ed458b855a2f44dc6d15ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:53.603343 2752 ts_tablet_manager.cc:1918] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:38:53.603413 2752 log.cc:1196] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-1/wal/wals/ad907f4844fa4797aa7fc4f74e0678c6
I20260927 16:38:53.603688 2752 ts_tablet_manager.cc:1939] T ad907f4844fa4797aa7fc4f74e0678c6 P 6de1875303d446e1841ca078453646ef: Deleting consensus metadata
I20260927 16:38:53.603802 2272 tablet_service.cc:1467] Processing CreateTablet for tablet 1183ef1121ed458b855a2f44dc6d15ad (DEFAULT_TABLE table=test-table [id=7e6b9a4fd86c40a5a0b6c1deae5fb4f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:38:53.604000 2272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1183ef1121ed458b855a2f44dc6d15ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:53.604298 2120 catalog_manager.cc:4841] TS 6de1875303d446e1841ca078453646ef (127.0.98.2:38761): tablet ad907f4844fa4797aa7fc4f74e0678c6 (table test-table [id=78ac7410e2d04ac69817b48e9d033fd5]) successfully deleted
I20260927 16:38:53.604106 2754 tablet_bootstrap.cc:492] T 1183ef1121ed458b855a2f44dc6d15ad P 01d50d647d674fd7a252b717c37cfc49: Bootstrap starting.
I20260927 16:38:53.604643 2755 tablet_bootstrap.cc:492] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef: Bootstrap starting.
I20260927 16:38:53.605365 2754 tablet_bootstrap.cc:654] T 1183ef1121ed458b855a2f44dc6d15ad P 01d50d647d674fd7a252b717c37cfc49: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:53.605715 2755 tablet_bootstrap.cc:654] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:53.606505 2757 tablet_bootstrap.cc:492] T 1183ef1121ed458b855a2f44dc6d15ad P b8c2d915da9f49abbac90af10ecd3337: Bootstrap starting.
I20260927 16:38:53.606900 2755 tablet_bootstrap.cc:492] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef: No bootstrap required, opened a new log
I20260927 16:38:53.606989 2755 ts_tablet_manager.cc:1397] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 16:38:53.607432 2757 tablet_bootstrap.cc:654] T 1183ef1121ed458b855a2f44dc6d15ad P b8c2d915da9f49abbac90af10ecd3337: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:53.607470 2755 raft_consensus.cc:348] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:53.607580 2755 raft_consensus.cc:374] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:53.607623 2755 raft_consensus.cc:729] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6de1875303d446e1841ca078453646ef, State: Initialized, Role: FOLLOWER
I20260927 16:38:53.607793 2755 consensus_queue.cc:260] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:53.608116 2754 tablet_bootstrap.cc:492] T 1183ef1121ed458b855a2f44dc6d15ad P 01d50d647d674fd7a252b717c37cfc49: No bootstrap required, opened a new log
I20260927 16:38:53.608125 2755 ts_tablet_manager.cc:1428] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:38:53.608248 2754 ts_tablet_manager.cc:1397] T 1183ef1121ed458b855a2f44dc6d15ad P 01d50d647d674fd7a252b717c37cfc49: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 16:38:53.608963 2754 raft_consensus.cc:348] T 1183ef1121ed458b855a2f44dc6d15ad P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:53.609126 2754 raft_consensus.cc:374] T 1183ef1121ed458b855a2f44dc6d15ad P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:53.609211 2754 raft_consensus.cc:729] T 1183ef1121ed458b855a2f44dc6d15ad P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d50d647d674fd7a252b717c37cfc49, State: Initialized, Role: FOLLOWER
I20260927 16:38:53.609385 2754 consensus_queue.cc:260] T 1183ef1121ed458b855a2f44dc6d15ad P 01d50d647d674fd7a252b717c37cfc49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:53.609678 2754 ts_tablet_manager.cc:1428] T 1183ef1121ed458b855a2f44dc6d15ad P 01d50d647d674fd7a252b717c37cfc49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:38:53.609535 2757 tablet_bootstrap.cc:492] T 1183ef1121ed458b855a2f44dc6d15ad P b8c2d915da9f49abbac90af10ecd3337: No bootstrap required, opened a new log
I20260927 16:38:53.609999 2757 ts_tablet_manager.cc:1397] T 1183ef1121ed458b855a2f44dc6d15ad P b8c2d915da9f49abbac90af10ecd3337: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 16:38:53.610425 2757 raft_consensus.cc:348] T 1183ef1121ed458b855a2f44dc6d15ad P b8c2d915da9f49abbac90af10ecd3337 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:53.610533 2757 raft_consensus.cc:374] T 1183ef1121ed458b855a2f44dc6d15ad P b8c2d915da9f49abbac90af10ecd3337 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:53.610569 2757 raft_consensus.cc:729] T 1183ef1121ed458b855a2f44dc6d15ad P b8c2d915da9f49abbac90af10ecd3337 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8c2d915da9f49abbac90af10ecd3337, State: Initialized, Role: FOLLOWER
I20260927 16:38:53.610702 2757 consensus_queue.cc:260] T 1183ef1121ed458b855a2f44dc6d15ad P b8c2d915da9f49abbac90af10ecd3337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:53.611425 2757 ts_tablet_manager.cc:1428] T 1183ef1121ed458b855a2f44dc6d15ad P b8c2d915da9f49abbac90af10ecd3337: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:38:53.794417 2731 raft_consensus.cc:482] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:38:53.794523 2731 raft_consensus.cc:504] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:53.794902 2731 leader_election.cc:288] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8c2d915da9f49abbac90af10ecd3337 (127.0.98.1:36215), 01d50d647d674fd7a252b717c37cfc49 (127.0.98.3:36137)
I20260927 16:38:53.795308 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1183ef1121ed458b855a2f44dc6d15ad" candidate_uuid: "6de1875303d446e1841ca078453646ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8c2d915da9f49abbac90af10ecd3337" is_pre_election: true
I20260927 16:38:53.795328 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1183ef1121ed458b855a2f44dc6d15ad" candidate_uuid: "6de1875303d446e1841ca078453646ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d50d647d674fd7a252b717c37cfc49" is_pre_election: true
I20260927 16:38:53.795439 2572 raft_consensus.cc:2454] T 1183ef1121ed458b855a2f44dc6d15ad P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6de1875303d446e1841ca078453646ef in term 0.
I20260927 16:38:53.795469 2291 raft_consensus.cc:2454] T 1183ef1121ed458b855a2f44dc6d15ad P b8c2d915da9f49abbac90af10ecd3337 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6de1875303d446e1841ca078453646ef in term 0.
I20260927 16:38:53.795737 2364 leader_election.cc:302] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d50d647d674fd7a252b717c37cfc49, 6de1875303d446e1841ca078453646ef; no voters:
I20260927 16:38:53.795941 2731 raft_consensus.cc:2781] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:38:53.796003 2731 raft_consensus.cc:482] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:38:53.796070 2731 raft_consensus.cc:3037] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:53.796973 2731 raft_consensus.cc:504] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:53.797288 2731 leader_election.cc:288] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef [CANDIDATE]: Term 1 election: Requested vote from peers b8c2d915da9f49abbac90af10ecd3337 (127.0.98.1:36215), 01d50d647d674fd7a252b717c37cfc49 (127.0.98.3:36137)
I20260927 16:38:53.797595 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1183ef1121ed458b855a2f44dc6d15ad" candidate_uuid: "6de1875303d446e1841ca078453646ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8c2d915da9f49abbac90af10ecd3337"
I20260927 16:38:53.797595 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1183ef1121ed458b855a2f44dc6d15ad" candidate_uuid: "6de1875303d446e1841ca078453646ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d50d647d674fd7a252b717c37cfc49"
I20260927 16:38:53.797701 2291 raft_consensus.cc:3037] T 1183ef1121ed458b855a2f44dc6d15ad P b8c2d915da9f49abbac90af10ecd3337 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:53.797701 2572 raft_consensus.cc:3037] T 1183ef1121ed458b855a2f44dc6d15ad P 01d50d647d674fd7a252b717c37cfc49 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:53.798805 2291 raft_consensus.cc:2454] T 1183ef1121ed458b855a2f44dc6d15ad P b8c2d915da9f49abbac90af10ecd3337 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6de1875303d446e1841ca078453646ef in term 1.
I20260927 16:38:53.798805 2572 raft_consensus.cc:2454] T 1183ef1121ed458b855a2f44dc6d15ad P 01d50d647d674fd7a252b717c37cfc49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6de1875303d446e1841ca078453646ef in term 1.
I20260927 16:38:53.799068 2364 leader_election.cc:302] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d50d647d674fd7a252b717c37cfc49, 6de1875303d446e1841ca078453646ef; no voters:
I20260927 16:38:53.799279 2731 raft_consensus.cc:2781] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:38:53.799407 2731 raft_consensus.cc:686] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef [term 1 LEADER]: Becoming Leader. State: Replica: 6de1875303d446e1841ca078453646ef, State: Running, Role: LEADER
I20260927 16:38:53.799547 2731 consensus_queue.cc:237] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } }
I20260927 16:38:53.801222 2134 catalog_manager.cc:5495] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 6de1875303d446e1841ca078453646ef (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6de1875303d446e1841ca078453646ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6de1875303d446e1841ca078453646ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 } health_report { overall_health: UNKNOWN } } }
I20260927 16:38:53.846243 2572 raft_consensus.cc:1261] T 1183ef1121ed458b855a2f44dc6d15ad P 01d50d647d674fd7a252b717c37cfc49 [term 1 FOLLOWER]: Refusing update from remote peer 6de1875303d446e1841ca078453646ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:38:53.846429 2291 raft_consensus.cc:1261] T 1183ef1121ed458b855a2f44dc6d15ad P b8c2d915da9f49abbac90af10ecd3337 [term 1 FOLLOWER]: Refusing update from remote peer 6de1875303d446e1841ca078453646ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:38:53.846690 2731 consensus_queue.cc:1035] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "01d50d647d674fd7a252b717c37cfc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:38:53.846850 2748 consensus_queue.cc:1035] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8c2d915da9f49abbac90af10ecd3337" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 16:38:53.859791 2697 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 16:38:55.403154 2134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:38:55.403338 2134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:38:55.405529 2134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2cc203aec9eb4fe4b7a505cacdf36c3f: Sending DeleteTablet for 3 replicas of tablet 1183ef1121ed458b855a2f44dc6d15ad
I20260927 16:38:55.406054 2412 tablet_service.cc:1514] Processing DeleteTablet for tablet 1183ef1121ed458b855a2f44dc6d15ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:38:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49228
I20260927 16:38:55.406122 2272 tablet_service.cc:1514] Processing DeleteTablet for tablet 1183ef1121ed458b855a2f44dc6d15ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:38:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33454
I20260927 16:38:55.406378 2784 tablet_replica.cc:330] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef: stopping tablet replica
I20260927 16:38:55.406416 2552 tablet_service.cc:1514] Processing DeleteTablet for tablet 1183ef1121ed458b855a2f44dc6d15ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:38:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37540
I20260927 16:38:55.406527 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 2207
I20260927 16:38:55.408021 2786 tablet_replica.cc:330] T 1183ef1121ed458b855a2f44dc6d15ad P 01d50d647d674fd7a252b717c37cfc49: stopping tablet replica
I20260927 16:38:55.408180 2786 raft_consensus.cc:2229] T 1183ef1121ed458b855a2f44dc6d15ad P 01d50d647d674fd7a252b717c37cfc49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:38:55.408293 2784 raft_consensus.cc:2229] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:38:55.408310 2786 raft_consensus.cc:2258] T 1183ef1121ed458b855a2f44dc6d15ad P 01d50d647d674fd7a252b717c37cfc49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:38:55.408470 2784 raft_consensus.cc:2258] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:38:55.409165 2784 ts_tablet_manager.cc:1905] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:38:55.409966 2786 ts_tablet_manager.cc:1905] T 1183ef1121ed458b855a2f44dc6d15ad P 01d50d647d674fd7a252b717c37cfc49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:38:55.411377 2784 ts_tablet_manager.cc:1918] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 16:38:55.411459 2784 log.cc:1196] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-1/wal/wals/1183ef1121ed458b855a2f44dc6d15ad
I20260927 16:38:55.411715 2784 ts_tablet_manager.cc:1939] T 1183ef1121ed458b855a2f44dc6d15ad P 6de1875303d446e1841ca078453646ef: Deleting consensus metadata
I20260927 16:38:55.412282 2120 catalog_manager.cc:4841] TS 6de1875303d446e1841ca078453646ef (127.0.98.2:38761): tablet 1183ef1121ed458b855a2f44dc6d15ad (table test-table [id=7e6b9a4fd86c40a5a0b6c1deae5fb4f0]) successfully deleted
I20260927 16:38:55.413187 2786 ts_tablet_manager.cc:1918] T 1183ef1121ed458b855a2f44dc6d15ad P 01d50d647d674fd7a252b717c37cfc49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 16:38:55.413280 2786 log.cc:1196] T 1183ef1121ed458b855a2f44dc6d15ad P 01d50d647d674fd7a252b717c37cfc49: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790527121515808-392-0/minicluster-data/ts-2/wal/wals/1183ef1121ed458b855a2f44dc6d15ad
I20260927 16:38:55.413543 2786 ts_tablet_manager.cc:1939] T 1183ef1121ed458b855a2f44dc6d15ad P 01d50d647d674fd7a252b717c37cfc49: Deleting consensus metadata
I20260927 16:38:55.414184 2120 catalog_manager.cc:4841] TS 01d50d647d674fd7a252b717c37cfc49 (127.0.98.3:36137): tablet 1183ef1121ed458b855a2f44dc6d15ad (table test-table [id=7e6b9a4fd86c40a5a0b6c1deae5fb4f0]) successfully deleted
W20260927 16:38:55.414318 2121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36215 (error 108)
I20260927 16:38:55.415048 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 2347
I20260927 16:38:55.416746 2788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:55.414916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36215 (local address 127.0.0.1:33460)
0927 16:38:55.416590 (+ 1674us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:55.416656 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":74,"threads_started":1}
W20260927 16:38:55.416852 2121 catalog_manager.cc:4584] TS b8c2d915da9f49abbac90af10ecd3337 (127.0.98.1:36215): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1183ef1121ed458b855a2f44dc6d15ad: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36215: connect: Connection refused (error 111)
I20260927 16:38:55.421281 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 2487
I20260927 16:38:55.427451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 2103
2026-09-27T16:38:55Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6583 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:38:55 dist-test-slave-z40b krb5kdc[2791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:38:55 dist-test-slave-z40b krb5kdc[2791](info): set up 2 sockets
Sep 27 16:38:55 dist-test-slave-z40b krb5kdc[2791](info): commencing operation
krb5kdc: starting...
W20260927 16:38:56.426940 392 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:38:56 dist-test-slave-z40b krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:38:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:38:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:38:56.538566 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33077
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:38:56.639364 2804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:38:56.639602 2804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:38:56.639693 2804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:38:56.643451 2804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:38:56.643524 2804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:38:56.643563 2804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:38:56.643615 2804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:38:56.643654 2804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:38:56.648483 2804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:38:56.649366 2804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:38:56.650254 2804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:38:56.658090 2810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:56.658105 2812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:56.658085 2809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:56.658329 2804 server_base.cc:1034] running on GCE node
I20260927 16:38:56.658845 2804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:38:56.659850 2804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:38:56.661023 2804 hybrid_clock.cc:648] HybridClock initialized: now 1790527136661011 us; error 24 us; skew 500 ppm
Sep 27 16:38:56 dist-test-slave-z40b krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:38:56.663523 2804 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:38:56.664285 2804 webserver.cc:460] Webserver started at http://127.0.98.62:38305/ using document root <none> and password file <none>
I20260927 16:38:56.664754 2804 fs_manager.cc:359] Metadata directory not provided
I20260927 16:38:56.664841 2804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:38:56.665060 2804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:38:56.666864 2804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "09b288f23b77429cbb9ae7f6b3d0b93f"
format_stamp: "Formatted at 2026-09-27 16:38:56 on dist-test-slave-z40b"
server_key: "f4a39dbe575fde70aae547974832f607"
server_key_iv: "d4c329630a534ae50a9d6fda2813cb80"
server_key_version: "encryptionkey@0"
I20260927 16:38:56.667397 2804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "09b288f23b77429cbb9ae7f6b3d0b93f"
format_stamp: "Formatted at 2026-09-27 16:38:56 on dist-test-slave-z40b"
server_key: "f4a39dbe575fde70aae547974832f607"
server_key_iv: "d4c329630a534ae50a9d6fda2813cb80"
server_key_version: "encryptionkey@0"
I20260927 16:38:56.670589 2804 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260927 16:38:56.672724 2819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:56.673779 2804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:38:56.673908 2804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "09b288f23b77429cbb9ae7f6b3d0b93f"
format_stamp: "Formatted at 2026-09-27 16:38:56 on dist-test-slave-z40b"
server_key: "f4a39dbe575fde70aae547974832f607"
server_key_iv: "d4c329630a534ae50a9d6fda2813cb80"
server_key_version: "encryptionkey@0"
I20260927 16:38:56.674005 2804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:38:56.683720 2804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:38:56.686542 2804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:38:56.686718 2804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:38:56.693490 2804 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33077
I20260927 16:38:56.693570 2871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33077 every 8 connection(s)
I20260927 16:38:56.694365 2804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:38:56.696939 2872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:56.701797 2872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09b288f23b77429cbb9ae7f6b3d0b93f: Bootstrap starting.
I20260927 16:38:56.703606 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 2804
I20260927 16:38:56.703715 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:38:56.703963 392 external_mini_cluster.cc:1428] Setting key de89b7947d75f45a80cf6dbd6218dc2d
I20260927 16:38:56.704236 2872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09b288f23b77429cbb9ae7f6b3d0b93f: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:56.705045 2872 log.cc:824] T 00000000000000000000000000000000 P 09b288f23b77429cbb9ae7f6b3d0b93f: Log is configured to *not* fsync() on all Append() calls
I20260927 16:38:56.707032 2872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09b288f23b77429cbb9ae7f6b3d0b93f: No bootstrap required, opened a new log
I20260927 16:38:56.711104 2872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09b288f23b77429cbb9ae7f6b3d0b93f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b288f23b77429cbb9ae7f6b3d0b93f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33077 } }
I20260927 16:38:56.711341 2872 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09b288f23b77429cbb9ae7f6b3d0b93f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:56.711409 2872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09b288f23b77429cbb9ae7f6b3d0b93f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09b288f23b77429cbb9ae7f6b3d0b93f, State: Initialized, Role: FOLLOWER
I20260927 16:38:56.711820 2872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09b288f23b77429cbb9ae7f6b3d0b93f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b288f23b77429cbb9ae7f6b3d0b93f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33077 } }
I20260927 16:38:56.711931 2872 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09b288f23b77429cbb9ae7f6b3d0b93f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:38:56.712026 2872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09b288f23b77429cbb9ae7f6b3d0b93f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:38:56.712155 2872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09b288f23b77429cbb9ae7f6b3d0b93f [term 0 FOLLOWER]: Advancing to term 1
Sep 27 16:38:56 dist-test-slave-z40b krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:38:56.713245 2872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09b288f23b77429cbb9ae7f6b3d0b93f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b288f23b77429cbb9ae7f6b3d0b93f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33077 } }
I20260927 16:38:56.713606 2872 leader_election.cc:302] T 00000000000000000000000000000000 P 09b288f23b77429cbb9ae7f6b3d0b93f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09b288f23b77429cbb9ae7f6b3d0b93f; no voters:
I20260927 16:38:56.713918 2872 leader_election.cc:288] T 00000000000000000000000000000000 P 09b288f23b77429cbb9ae7f6b3d0b93f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:38:56.714027 2877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09b288f23b77429cbb9ae7f6b3d0b93f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:38:56.714278 2877 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09b288f23b77429cbb9ae7f6b3d0b93f [term 1 LEADER]: Becoming Leader. State: Replica: 09b288f23b77429cbb9ae7f6b3d0b93f, State: Running, Role: LEADER
I20260927 16:38:56.714644 2877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09b288f23b77429cbb9ae7f6b3d0b93f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b288f23b77429cbb9ae7f6b3d0b93f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33077 } }
I20260927 16:38:56.715091 2872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09b288f23b77429cbb9ae7f6b3d0b93f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:38:56.716423 2878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09b288f23b77429cbb9ae7f6b3d0b93f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09b288f23b77429cbb9ae7f6b3d0b93f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b288f23b77429cbb9ae7f6b3d0b93f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33077 } } }
I20260927 16:38:56.716569 2878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09b288f23b77429cbb9ae7f6b3d0b93f [sys.catalog]: This master's current role is: LEADER
I20260927 16:38:56.716791 2878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09b288f23b77429cbb9ae7f6b3d0b93f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09b288f23b77429cbb9ae7f6b3d0b93f. Latest consensus state: current_term: 1 leader_uuid: "09b288f23b77429cbb9ae7f6b3d0b93f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b288f23b77429cbb9ae7f6b3d0b93f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33077 } } }
I20260927 16:38:56.716889 2878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09b288f23b77429cbb9ae7f6b3d0b93f [sys.catalog]: This master's current role is: LEADER
I20260927 16:38:56.719558 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:56.704959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44300 (local address 127.0.98.62:33077)
0927 16:38:56.705365 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:56.705386 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:56.705428 (+ 42us) server_negotiation.cc:407] Connection header received
0927 16:38:56.706184 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:56.706206 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:56.706582 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:56.706910 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:56.707664 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:56.709116 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:56.709917 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:56.710624 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:56.710776 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:56.713665 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:56.713670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:56.713681 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:56.713720 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:56.715973 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:56.716565 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:56.716570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:56.716577 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:56.716745 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:56.717087 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:56.717107 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:56.717111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:56.717571 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:56.717778 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:56.718343 (+ 565us) server_negotiation.cc:299] Negotiation successful
0927 16:38:56.718652 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":119,"threads_started":1}
W20260927 16:38:56.719925 2892 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09b288f23b77429cbb9ae7f6b3d0b93f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:38:56.720031 2892 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:38:56.720134 2889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:38:56.720886 2889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:38:56.725327 2889 catalog_manager.cc:1347] Generated new cluster ID: 7d9309d9259f414ea989b1bb4ab1ba39
I20260927 16:38:56.725385 2889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:38:56.732671 2889 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:38:56.733448 2889 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:38:56.740540 2889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09b288f23b77429cbb9ae7f6b3d0b93f: Generated new TSK 0
I20260927 16:38:56.741071 2889 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:38:56.770627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33077
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:38:56.884099 2898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:38:56.884316 2898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:38:56.884374 2898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:38:56.888242 2898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:38:56.888295 2898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:38:56.888387 2898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:38:56.893018 2898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:38:56.893831 2898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:38:56.894707 2898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:38:56.903330 2906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:56.903373 2898 server_base.cc:1034] running on GCE node
W20260927 16:38:56.903442 2904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:56.903328 2903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:56.903877 2898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:38:56.904426 2898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:38:56.905591 2898 hybrid_clock.cc:648] HybridClock initialized: now 1790527136905565 us; error 38 us; skew 500 ppm
Sep 27 16:38:56 dist-test-slave-z40b krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:38:56.908702 2898 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:38:56.909412 2898 webserver.cc:460] Webserver started at http://127.0.98.1:40871/ using document root <none> and password file <none>
I20260927 16:38:56.909875 2898 fs_manager.cc:359] Metadata directory not provided
I20260927 16:38:56.909926 2898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:38:56.910066 2898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:38:56.911942 2898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/data/instance:
uuid: "13cbcb85195d43e1ad6591247ec1cc44"
format_stamp: "Formatted at 2026-09-27 16:38:56 on dist-test-slave-z40b"
server_key: "c2b1b71438284245dd3479dd5e65943d"
server_key_iv: "3d6db61347c6b24c2e65bd7e803ccd1d"
server_key_version: "encryptionkey@0"
I20260927 16:38:56.912410 2898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/wal/instance:
uuid: "13cbcb85195d43e1ad6591247ec1cc44"
format_stamp: "Formatted at 2026-09-27 16:38:56 on dist-test-slave-z40b"
server_key: "c2b1b71438284245dd3479dd5e65943d"
server_key_iv: "3d6db61347c6b24c2e65bd7e803ccd1d"
server_key_version: "encryptionkey@0"
I20260927 16:38:56.915556 2898 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:38:56.917675 2913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:56.918758 2898 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:38:56.918867 2898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "13cbcb85195d43e1ad6591247ec1cc44"
format_stamp: "Formatted at 2026-09-27 16:38:56 on dist-test-slave-z40b"
server_key: "c2b1b71438284245dd3479dd5e65943d"
server_key_iv: "3d6db61347c6b24c2e65bd7e803ccd1d"
server_key_version: "encryptionkey@0"
I20260927 16:38:56.918956 2898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:38:56.933313 2898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:38:56.936180 2898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:38:56.936360 2898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:38:56.936841 2898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:38:56.937518 2898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:38:56.937561 2898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:56.937623 2898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:38:56.937650 2898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:56.945549 2898 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37411
I20260927 16:38:56.945574 3026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37411 every 8 connection(s)
I20260927 16:38:56.946327 2898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:38:56.948945 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 2898
I20260927 16:38:56.949059 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:38:56.949380 392 external_mini_cluster.cc:1428] Setting key e89b9d3e1202686ff71e53f7744fbe17
Sep 27 16:38:56 dist-test-slave-z40b krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:38:56.959753 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:56.947875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45731 (local address 127.0.98.62:33077)
0927 16:38:56.948007 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:56.948011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:56.948748 (+ 737us) server_negotiation.cc:407] Connection header received
0927 16:38:56.950299 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:56.950303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:56.950355 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:56.950448 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:56.951855 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:56.952551 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:56.953728 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:56.953974 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:56.954081 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:56.956420 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:56.956424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:56.956431 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:56.956451 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:56.957816 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:56.958306 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:56.958317 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:56.958319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:56.958392 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:56.958715 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:56.958718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:56.958720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:56.958885 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:56.958985 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:56.959474 (+ 489us) server_negotiation.cc:299] Negotiation successful
0927 16:38:56.959601 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 16:38:56.960445 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:56.948183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33077 (local address 127.0.98.1:45731)
0927 16:38:56.948590 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:56.948627 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:56.950108 (+ 1481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:56.950598 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:56.950605 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:56.951039 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:56.951693 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:56.951724 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:56.952749 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:56.952751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:56.953594 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:56.953602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:56.954109 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:56.954113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:56.954301 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:56.954867 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:56.954882 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:56.956257 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:56.957934 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:56.957939 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:56.957952 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:56.958215 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:56.958487 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:56.958490 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:56.958492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:56.958619 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:56.958983 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:56.958990 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:56.959241 (+ 251us) client_negotiation.cc:769] Sending connection context
0927 16:38:56.959433 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 16:38:56.959713 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":93,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 16:38:56.961591 3027 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33077
I20260927 16:38:56.961814 3027 heartbeater.cc:464] Registering TS with master...
I20260927 16:38:56.962342 3027 heartbeater.cc:511] Master 127.0.98.62:33077 requested a full tablet report, sending...
I20260927 16:38:56.963722 2836 ts_manager.cc:194] Registered new tserver with Master: 13cbcb85195d43e1ad6591247ec1cc44 (127.0.98.1:37411)
I20260927 16:38:56.964704 2836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45731
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:38:56.974367 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33077
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:38:57.076870 3032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:38:57.077080 3032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:38:57.077127 3032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:38:57.080977 3032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:38:57.081019 3032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:38:57.081089 3032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:38:57.085883 3032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:38:57.086709 3032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:38:57.087602 3032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:38:57.095788 3038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:57.095785 3040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:57.095773 3037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:57.096016 3032 server_base.cc:1034] running on GCE node
I20260927 16:38:57.096441 3032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:38:57.096936 3032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:38:57.098078 3032 hybrid_clock.cc:648] HybridClock initialized: now 1790527137098069 us; error 21 us; skew 500 ppm
Sep 27 16:38:57 dist-test-slave-z40b krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:38:57.100955 3032 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:38:57.101747 3032 webserver.cc:460] Webserver started at http://127.0.98.2:43483/ using document root <none> and password file <none>
I20260927 16:38:57.102238 3032 fs_manager.cc:359] Metadata directory not provided
I20260927 16:38:57.102296 3032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:38:57.102509 3032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:38:57.104333 3032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/data/instance:
uuid: "2491c5e47a75439d85dec3be6f506e28"
format_stamp: "Formatted at 2026-09-27 16:38:57 on dist-test-slave-z40b"
server_key: "851f22b29bafb7413fb47b76f311cedc"
server_key_iv: "7c581321d1233ba462952815775f956f"
server_key_version: "encryptionkey@0"
I20260927 16:38:57.104842 3032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2491c5e47a75439d85dec3be6f506e28"
format_stamp: "Formatted at 2026-09-27 16:38:57 on dist-test-slave-z40b"
server_key: "851f22b29bafb7413fb47b76f311cedc"
server_key_iv: "7c581321d1233ba462952815775f956f"
server_key_version: "encryptionkey@0"
I20260927 16:38:57.108281 3032 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:38:57.110448 3047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:57.111629 3032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:38:57.111738 3032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "2491c5e47a75439d85dec3be6f506e28"
format_stamp: "Formatted at 2026-09-27 16:38:57 on dist-test-slave-z40b"
server_key: "851f22b29bafb7413fb47b76f311cedc"
server_key_iv: "7c581321d1233ba462952815775f956f"
server_key_version: "encryptionkey@0"
I20260927 16:38:57.111827 3032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:38:57.134116 3032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:38:57.136672 3032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:38:57.136866 3032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:38:57.137343 3032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:38:57.138017 3032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:38:57.138056 3032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:57.138142 3032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:38:57.138175 3032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:57.146111 3032 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44901
I20260927 16:38:57.146147 3160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44901 every 8 connection(s)
I20260927 16:38:57.146931 3032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:38:57.149683 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 3032
I20260927 16:38:57.149792 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:38:57.150035 392 external_mini_cluster.cc:1428] Setting key af350898b1859d6b159e515cd93be4f6
Sep 27 16:38:57 dist-test-slave-z40b krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 16:38:57.160084 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:57.148583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45117 (local address 127.0.98.62:33077)
0927 16:38:57.148711 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:57.148715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:57.149436 (+ 721us) server_negotiation.cc:407] Connection header received
0927 16:38:57.150937 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:57.150940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:57.150979 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:57.151049 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:57.152167 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:57.152727 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:57.153776 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:57.154161 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:57.154267 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:57.156781 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:57.156784 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:57.156786 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:57.156806 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:57.158184 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:57.158765 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:57.158768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:57.158769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:57.158840 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:57.159122 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:57.159124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:57.159125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:57.159347 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:57.159424 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:57.159869 (+ 445us) server_negotiation.cc:299] Negotiation successful
0927 16:38:57.159965 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:38:57.160784 3163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:57.148863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33077 (local address 127.0.98.2:45117)
0927 16:38:57.149274 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:57.149312 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:57.150752 (+ 1440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:57.151225 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:57.151248 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:57.151633 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:57.152036 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:57.152047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:57.152860 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:57.152862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:57.153660 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:57.153669 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:57.154320 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:57.154322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:57.154442 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:57.154949 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:57.154963 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:57.156627 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:57.158294 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:57.158299 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:57.158312 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:57.158669 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:57.158923 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:57.158926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:57.158927 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:57.159047 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:57.159431 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:57.159435 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:57.159673 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 16:38:57.159827 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 16:38:57.160036 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 16:38:57.161877 3161 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33077
I20260927 16:38:57.162101 3161 heartbeater.cc:464] Registering TS with master...
I20260927 16:38:57.162655 3161 heartbeater.cc:511] Master 127.0.98.62:33077 requested a full tablet report, sending...
I20260927 16:38:57.163687 2836 ts_manager.cc:194] Registered new tserver with Master: 2491c5e47a75439d85dec3be6f506e28 (127.0.98.2:44901)
I20260927 16:38:57.164191 2836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45117
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:38:57.174015 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33077
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:38:57.275341 3166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:38:57.275538 3166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:38:57.275575 3166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:38:57.279778 3166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:38:57.279824 3166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:38:57.279879 3166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:38:57.284530 3166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:38:57.285360 3166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:38:57.286193 3166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:38:57.295323 3172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:57.295331 3171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:57.295377 3174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:57.295495 3166 server_base.cc:1034] running on GCE node
I20260927 16:38:57.296015 3166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:38:57.296545 3166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:38:57.297709 3166 hybrid_clock.cc:648] HybridClock initialized: now 1790527137297682 us; error 45 us; skew 500 ppm
Sep 27 16:38:57 dist-test-slave-z40b krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:38:57.300140 3166 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:38:57.300886 3166 webserver.cc:460] Webserver started at http://127.0.98.3:40081/ using document root <none> and password file <none>
I20260927 16:38:57.301342 3166 fs_manager.cc:359] Metadata directory not provided
I20260927 16:38:57.301396 3166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:38:57.301540 3166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:38:57.303347 3166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/data/instance:
uuid: "2991efea502646649ffb401eee6f8340"
format_stamp: "Formatted at 2026-09-27 16:38:57 on dist-test-slave-z40b"
server_key: "35d72f3c8913cddb8ec5ea469ef34b7f"
server_key_iv: "e4b56c0218a972972112c213278f37f8"
server_key_version: "encryptionkey@0"
I20260927 16:38:57.303823 3166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2991efea502646649ffb401eee6f8340"
format_stamp: "Formatted at 2026-09-27 16:38:57 on dist-test-slave-z40b"
server_key: "35d72f3c8913cddb8ec5ea469ef34b7f"
server_key_iv: "e4b56c0218a972972112c213278f37f8"
server_key_version: "encryptionkey@0"
I20260927 16:38:57.307087 3166 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:38:57.309170 3181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:57.310252 3166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:38:57.310343 3166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "2991efea502646649ffb401eee6f8340"
format_stamp: "Formatted at 2026-09-27 16:38:57 on dist-test-slave-z40b"
server_key: "35d72f3c8913cddb8ec5ea469ef34b7f"
server_key_iv: "e4b56c0218a972972112c213278f37f8"
server_key_version: "encryptionkey@0"
I20260927 16:38:57.310408 3166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:38:57.329149 3166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:38:57.331712 3166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:38:57.331846 3166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:38:57.332299 3166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:38:57.332993 3166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:38:57.333035 3166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:57.333071 3166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:38:57.333092 3166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:57.341122 3166 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38967
I20260927 16:38:57.341150 3294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38967 every 8 connection(s)
I20260927 16:38:57.341889 3166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:38:57.348819 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 3166
I20260927 16:38:57.348938 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:38:57.349191 392 external_mini_cluster.cc:1428] Setting key 1ffd0516a339e7f1a4efc06cb4d96155
Sep 27 16:38:57 dist-test-slave-z40b krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:38:57.354255 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:57.343561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43731 (local address 127.0.98.62:33077)
0927 16:38:57.343694 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:57.343697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:57.344411 (+ 714us) server_negotiation.cc:407] Connection header received
0927 16:38:57.345747 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:57.345751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:57.345806 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:57.345910 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:57.346981 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:57.347547 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:57.348619 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:57.348850 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:57.348959 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:57.351242 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:57.351245 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:57.351248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:57.351276 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:57.352423 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:57.352882 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:57.352887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:57.352889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:57.352963 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:57.353261 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:57.353266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:57.353267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:57.353435 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:57.353532 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:57.353986 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 16:38:57.354096 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:38:57.354934 3297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:57.343897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33077 (local address 127.0.98.3:43731)
0927 16:38:57.344281 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0927 16:38:57.344318 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 16:38:57.345549 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:38:57.346074 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:38:57.346082 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:38:57.346398 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:38:57.346816 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:57.346857 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:57.347658 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:57.347662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:57.348497 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:38:57.348506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:38:57.348987 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:38:57.348992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:38:57.349147 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:57.349717 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:38:57.349740 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:38:57.351077 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:38:57.352531 (+ 1454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:57.352536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:57.352548 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:57.352780 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:57.353047 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:38:57.353051 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:38:57.353053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:38:57.353173 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:38:57.353530 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:38:57.353536 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:38:57.353767 (+ 231us) client_negotiation.cc:769] Sending connection context
0927 16:38:57.353942 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 16:38:57.354220 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":96,"threads_started":1}
I20260927 16:38:57.355975 3295 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33077
I20260927 16:38:57.356158 3295 heartbeater.cc:464] Registering TS with master...
I20260927 16:38:57.356528 3295 heartbeater.cc:511] Master 127.0.98.62:33077 requested a full tablet report, sending...
I20260927 16:38:57.357426 2836 ts_manager.cc:194] Registered new tserver with Master: 2991efea502646649ffb401eee6f8340 (127.0.98.3:38967)
I20260927 16:38:57.357914 2836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43731
I20260927 16:38:57.362707 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 16:38:57.374334 2875 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:38:57.373148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44304 (local address 127.0.98.62:33077)
0927 16:38:57.373247 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:57.373250 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:57.373438 (+ 188us) server_negotiation.cc:407] Connection header received
0927 16:38:57.373557 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:57.373559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:57.373593 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:57.373660 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:57.374209 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44304: BlockingRecv error: recv got EOF from 127.0.0.1:44304 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 16:38:57.375800 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 2898
I20260927 16:38:57.380880 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 3032
I20260927 16:38:57.385578 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 3166
I20260927 16:38:57.390482 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 2804
2026-09-27T16:38:57Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1961 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:38:57 dist-test-slave-z40b krb5kdc[3307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:38:57 dist-test-slave-z40b krb5kdc[3307](info): set up 2 sockets
Sep 27 16:38:57 dist-test-slave-z40b krb5kdc[3307](info): commencing operation
krb5kdc: starting...
W20260927 16:38:58.411230 392 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:38:58 dist-test-slave-z40b krb5kdc[3307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:38:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:38:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:38:58.524117 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43383
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:38:58.629792 3320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:38:58.630048 3320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:38:58.630116 3320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:38:58.633805 3320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:38:58.633873 3320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:38:58.633909 3320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:38:58.633956 3320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:38:58.633992 3320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:38:58.639333 3320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:38:58.640202 3320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:38:58.641104 3320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:38:58.648806 3326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:58.648916 3320 server_base.cc:1034] running on GCE node
W20260927 16:38:58.648794 3328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:38:58.648834 3325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:38:58.649477 3320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:38:58.650425 3320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:38:58.651599 3320 hybrid_clock.cc:648] HybridClock initialized: now 1790527138651566 us; error 43 us; skew 500 ppm
Sep 27 16:38:58 dist-test-slave-z40b krb5kdc[3307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:38:58.657346 3320 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:38:58.658063 3320 webserver.cc:460] Webserver started at http://127.0.98.62:35011/ using document root <none> and password file <none>
I20260927 16:38:58.658593 3320 fs_manager.cc:359] Metadata directory not provided
I20260927 16:38:58.658668 3320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:38:58.658818 3320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:38:58.660742 3320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "68325b93e6544cf7b7c08b1b27b236b1"
format_stamp: "Formatted at 2026-09-27 16:38:58 on dist-test-slave-z40b"
server_key: "3efe656818241b69a052dba9dd36f8ef"
server_key_iv: "7b77b72adc4dda1daa9dcc5057a46017"
server_key_version: "encryptionkey@0"
I20260927 16:38:58.661201 3320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "68325b93e6544cf7b7c08b1b27b236b1"
format_stamp: "Formatted at 2026-09-27 16:38:58 on dist-test-slave-z40b"
server_key: "3efe656818241b69a052dba9dd36f8ef"
server_key_iv: "7b77b72adc4dda1daa9dcc5057a46017"
server_key_version: "encryptionkey@0"
I20260927 16:38:58.664433 3320 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:38:58.666531 3335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:38:58.667793 3320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:38:58.667877 3320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "68325b93e6544cf7b7c08b1b27b236b1"
format_stamp: "Formatted at 2026-09-27 16:38:58 on dist-test-slave-z40b"
server_key: "3efe656818241b69a052dba9dd36f8ef"
server_key_iv: "7b77b72adc4dda1daa9dcc5057a46017"
server_key_version: "encryptionkey@0"
I20260927 16:38:58.667948 3320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:38:58.690616 3320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:38:58.693415 3320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:38:58.693547 3320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:38:58.700060 3320 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43383
I20260927 16:38:58.700316 3387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43383 every 8 connection(s)
I20260927 16:38:58.701014 3320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:38:58.703483 3388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:38:58.708644 3388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68325b93e6544cf7b7c08b1b27b236b1: Bootstrap starting.
I20260927 16:38:58.709040 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 3320
I20260927 16:38:58.709175 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:38:58.709438 392 external_mini_cluster.cc:1428] Setting key 14d44f42320e31438a78f183f71cd2c5
I20260927 16:38:58.711185 3388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68325b93e6544cf7b7c08b1b27b236b1: Neither blocks nor log segments found. Creating new log.
I20260927 16:38:58.711968 3388 log.cc:824] T 00000000000000000000000000000000 P 68325b93e6544cf7b7c08b1b27b236b1: Log is configured to *not* fsync() on all Append() calls
I20260927 16:38:58.713625 3388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68325b93e6544cf7b7c08b1b27b236b1: No bootstrap required, opened a new log
I20260927 16:38:58.716845 3388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68325b93e6544cf7b7c08b1b27b236b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68325b93e6544cf7b7c08b1b27b236b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43383 } }
I20260927 16:38:58.717056 3388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68325b93e6544cf7b7c08b1b27b236b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:38:58.717127 3388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68325b93e6544cf7b7c08b1b27b236b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68325b93e6544cf7b7c08b1b27b236b1, State: Initialized, Role: FOLLOWER
I20260927 16:38:58.717519 3388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68325b93e6544cf7b7c08b1b27b236b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68325b93e6544cf7b7c08b1b27b236b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43383 } }
Sep 27 16:38:58 dist-test-slave-z40b krb5kdc[3307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:38:58.717669 3388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68325b93e6544cf7b7c08b1b27b236b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:38:58.717747 3388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68325b93e6544cf7b7c08b1b27b236b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:38:58.717842 3388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68325b93e6544cf7b7c08b1b27b236b1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:38:58.718611 3388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68325b93e6544cf7b7c08b1b27b236b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68325b93e6544cf7b7c08b1b27b236b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43383 } }
I20260927 16:38:58.718938 3388 leader_election.cc:302] T 00000000000000000000000000000000 P 68325b93e6544cf7b7c08b1b27b236b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68325b93e6544cf7b7c08b1b27b236b1; no voters:
I20260927 16:38:58.719231 3388 leader_election.cc:288] T 00000000000000000000000000000000 P 68325b93e6544cf7b7c08b1b27b236b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:38:58.719578 3393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68325b93e6544cf7b7c08b1b27b236b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:38:58.719808 3393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68325b93e6544cf7b7c08b1b27b236b1 [term 1 LEADER]: Becoming Leader. State: Replica: 68325b93e6544cf7b7c08b1b27b236b1, State: Running, Role: LEADER
I20260927 16:38:58.720144 3393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68325b93e6544cf7b7c08b1b27b236b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68325b93e6544cf7b7c08b1b27b236b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43383 } }
I20260927 16:38:58.720634 3388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68325b93e6544cf7b7c08b1b27b236b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:38:58.721293 3393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68325b93e6544cf7b7c08b1b27b236b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68325b93e6544cf7b7c08b1b27b236b1. Latest consensus state: current_term: 1 leader_uuid: "68325b93e6544cf7b7c08b1b27b236b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68325b93e6544cf7b7c08b1b27b236b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43383 } } }
I20260927 16:38:58.721472 3393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68325b93e6544cf7b7c08b1b27b236b1 [sys.catalog]: This master's current role is: LEADER
I20260927 16:38:58.721872 3402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:38:58.722386 3395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68325b93e6544cf7b7c08b1b27b236b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68325b93e6544cf7b7c08b1b27b236b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68325b93e6544cf7b7c08b1b27b236b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43383 } } }
I20260927 16:38:58.722568 3395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68325b93e6544cf7b7c08b1b27b236b1 [sys.catalog]: This master's current role is: LEADER
I20260927 16:38:58.723877 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:58.710648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46704 (local address 127.0.98.62:43383)
0927 16:38:58.711039 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:58.711052 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:58.711084 (+ 32us) server_negotiation.cc:407] Connection header received
0927 16:38:58.711792 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:58.711808 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:58.712156 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:58.712458 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:58.713223 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:58.714216 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:58.715060 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:58.715555 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:58.715710 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:58.718215 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:58.718220 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:58.718232 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:38:58.718256 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:58.720341 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:58.720804 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:58.720809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:58.720814 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:58.720984 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:38:58.721301 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:38:58.721307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:38:58.721311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:38:58.721801 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:58.721971 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:58.722748 (+ 777us) server_negotiation.cc:299] Negotiation successful
0927 16:38:58.723020 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":118,"threads_started":1}
I20260927 16:38:58.724790 3402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:38:58.726823 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:38:58.730300 3402 catalog_manager.cc:1347] Generated new cluster ID: 5fa5a5351f8b4852ad384b4a3238b127
I20260927 16:38:58.730374 3402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:38:58.743327 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:38:58.739192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46720 (local address 127.0.98.62:43383)
0927 16:38:58.739344 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 16:38:58.739349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:38:58.739565 (+ 216us) server_negotiation.cc:407] Connection header received
0927 16:38:58.739711 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:38:58.739714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:38:58.739757 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:38:58.739860 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:38:58.740285 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:58.740731 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:58.741481 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:38:58.741687 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:38:58.741785 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:38:58.742486 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:38:58.742490 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:38:58.742492 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:38:58.742886 (+ 394us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:38:58.742909 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:38:58.742924 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:38:58.743015 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 16:38:58.743092 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 16:38:58.743142 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 16:38:58.755699 3402 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:38:58.756444 3402 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:38:58.767448 3402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68325b93e6544cf7b7c08b1b27b236b1: Generated new TSK 0
I20260927 16:38:58.768021 3402 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:38:58.779618 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 3320
2026-09-27T16:38:58Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1388 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:38:58 dist-test-slave-z40b krb5kdc[3421](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:38:58 dist-test-slave-z40b krb5kdc[3421](info): set up 2 sockets
Sep 27 16:38:58 dist-test-slave-z40b krb5kdc[3421](info): commencing operation
krb5kdc: starting...
W20260927 16:38:59.778924 392 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:38:59 dist-test-slave-z40b krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:38:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:38:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:38:59.889994 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45169
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45169
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:00.001718 3434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:00.002017 3434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:00.002092 3434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:00.006176 3434 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:39:00.006258 3434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:00.006304 3434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:00.006354 3434 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:39:00.006393 3434 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:39:00.011587 3434 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:00.012663 3434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:00.013712 3434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:00.021634 3442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:00.021728 3439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:00.021766 3440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:00.021797 3434 server_base.cc:1034] running on GCE node
I20260927 16:39:00.022358 3434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:00.023474 3434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:00.024628 3434 hybrid_clock.cc:648] HybridClock initialized: now 1790527140024616 us; error 23 us; skew 500 ppm
Sep 27 16:39:00 dist-test-slave-z40b krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:00.027333 3434 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:39:00.028115 3434 webserver.cc:460] Webserver started at http://127.0.98.62:44537/ using document root <none> and password file <none>
I20260927 16:39:00.028594 3434 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:00.028647 3434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:00.028812 3434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:00.030638 3434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "881684182e7740b3b11af16ae953f5bf"
format_stamp: "Formatted at 2026-09-27 16:39:00 on dist-test-slave-z40b"
server_key: "bae75649496a85f0b7a68519ded53b36"
server_key_iv: "82b6446804e963e9a611c96a6997389d"
server_key_version: "encryptionkey@0"
I20260927 16:39:00.031126 3434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "881684182e7740b3b11af16ae953f5bf"
format_stamp: "Formatted at 2026-09-27 16:39:00 on dist-test-slave-z40b"
server_key: "bae75649496a85f0b7a68519ded53b36"
server_key_iv: "82b6446804e963e9a611c96a6997389d"
server_key_version: "encryptionkey@0"
I20260927 16:39:00.034775 3434 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:39:00.036974 3449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:00.038072 3434 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:00.038205 3434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "881684182e7740b3b11af16ae953f5bf"
format_stamp: "Formatted at 2026-09-27 16:39:00 on dist-test-slave-z40b"
server_key: "bae75649496a85f0b7a68519ded53b36"
server_key_iv: "82b6446804e963e9a611c96a6997389d"
server_key_version: "encryptionkey@0"
I20260927 16:39:00.038305 3434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:00.054471 3434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:00.057572 3434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:00.057768 3434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:00.064230 3434 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45169
I20260927 16:39:00.064507 3501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45169 every 8 connection(s)
I20260927 16:39:00.065176 3434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:39:00.067692 3502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:00.072750 3502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881684182e7740b3b11af16ae953f5bf: Bootstrap starting.
I20260927 16:39:00.074884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 3434
I20260927 16:39:00.074922 3502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 881684182e7740b3b11af16ae953f5bf: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:00.074996 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:39:00.075281 392 external_mini_cluster.cc:1428] Setting key 90cd7c636340afda9d8caf33f4ff111c
I20260927 16:39:00.075663 3502 log.cc:824] T 00000000000000000000000000000000 P 881684182e7740b3b11af16ae953f5bf: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:00.078202 3502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881684182e7740b3b11af16ae953f5bf: No bootstrap required, opened a new log
I20260927 16:39:00.082270 3502 raft_consensus.cc:348] T 00000000000000000000000000000000 P 881684182e7740b3b11af16ae953f5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881684182e7740b3b11af16ae953f5bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45169 } }
I20260927 16:39:00.082480 3502 raft_consensus.cc:374] T 00000000000000000000000000000000 P 881684182e7740b3b11af16ae953f5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:00.082551 3502 raft_consensus.cc:729] T 00000000000000000000000000000000 P 881684182e7740b3b11af16ae953f5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881684182e7740b3b11af16ae953f5bf, State: Initialized, Role: FOLLOWER
I20260927 16:39:00.082940 3502 consensus_queue.cc:260] T 00000000000000000000000000000000 P 881684182e7740b3b11af16ae953f5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881684182e7740b3b11af16ae953f5bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45169 } }
I20260927 16:39:00.083076 3502 raft_consensus.cc:388] T 00000000000000000000000000000000 P 881684182e7740b3b11af16ae953f5bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:39:00.083150 3502 raft_consensus.cc:482] T 00000000000000000000000000000000 P 881684182e7740b3b11af16ae953f5bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:39:00.083282 3502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 881684182e7740b3b11af16ae953f5bf [term 0 FOLLOWER]: Advancing to term 1
Sep 27 16:39:00 dist-test-slave-z40b krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:39:00.084071 3502 raft_consensus.cc:504] T 00000000000000000000000000000000 P 881684182e7740b3b11af16ae953f5bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881684182e7740b3b11af16ae953f5bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45169 } }
I20260927 16:39:00.084401 3502 leader_election.cc:302] T 00000000000000000000000000000000 P 881684182e7740b3b11af16ae953f5bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 881684182e7740b3b11af16ae953f5bf; no voters:
I20260927 16:39:00.084712 3502 leader_election.cc:288] T 00000000000000000000000000000000 P 881684182e7740b3b11af16ae953f5bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:39:00.084777 3507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 881684182e7740b3b11af16ae953f5bf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:39:00.085005 3507 raft_consensus.cc:686] T 00000000000000000000000000000000 P 881684182e7740b3b11af16ae953f5bf [term 1 LEADER]: Becoming Leader. State: Replica: 881684182e7740b3b11af16ae953f5bf, State: Running, Role: LEADER
I20260927 16:39:00.085330 3507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 881684182e7740b3b11af16ae953f5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881684182e7740b3b11af16ae953f5bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45169 } }
I20260927 16:39:00.085903 3502 sys_catalog.cc:564] T 00000000000000000000000000000000 P 881684182e7740b3b11af16ae953f5bf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:39:00.086912 3508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881684182e7740b3b11af16ae953f5bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "881684182e7740b3b11af16ae953f5bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881684182e7740b3b11af16ae953f5bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45169 } } }
I20260927 16:39:00.087038 3508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881684182e7740b3b11af16ae953f5bf [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:00.087306 3509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881684182e7740b3b11af16ae953f5bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 881684182e7740b3b11af16ae953f5bf. Latest consensus state: current_term: 1 leader_uuid: "881684182e7740b3b11af16ae953f5bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881684182e7740b3b11af16ae953f5bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45169 } } }
I20260927 16:39:00.087396 3509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881684182e7740b3b11af16ae953f5bf [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:00.087445 3516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:39:00.089713 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:00.076315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51816 (local address 127.0.98.62:45169)
0927 16:39:00.076759 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:00.076771 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:00.076800 (+ 29us) server_negotiation.cc:407] Connection header received
0927 16:39:00.077574 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:00.077594 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:00.077951 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:00.078264 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:00.078913 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:00.080411 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:00.081241 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:00.081910 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:00.082058 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:00.084607 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:00.084611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:00.084623 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:00.084647 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:00.086546 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:00.087129 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:00.087135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:00.087140 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:00.087309 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:00.087643 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:00.087646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:00.087649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:00.088124 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:00.088336 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:00.088589 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 16:39:00.088923 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":111,"threads_started":1}
I20260927 16:39:00.090528 3516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:39:00.095749 3516 catalog_manager.cc:1347] Generated new cluster ID: f376b95af9fe4ac5bd636d7a0c460195
I20260927 16:39:00.095810 3516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:39:00.132584 3516 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:39:00.133724 3516 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:39:00.141762 3516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 881684182e7740b3b11af16ae953f5bf: Generated new TSK 0
I20260927 16:39:00.142400 3516 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:39:00.179898 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45169
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:00.282567 3528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:00.282795 3528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:00.282879 3528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:00.286710 3528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:00.286770 3528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:00.286844 3528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:39:00.291810 3528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:00.292697 3528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:00.293579 3528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:00.302111 3534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:00.302124 3533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:00.302165 3528 server_base.cc:1034] running on GCE node
W20260927 16:39:00.302169 3536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:00.302649 3528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:00.303156 3528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:00.304334 3528 hybrid_clock.cc:648] HybridClock initialized: now 1790527140304314 us; error 31 us; skew 500 ppm
Sep 27 16:39:00 dist-test-slave-z40b krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:00.306829 3528 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:39:00.307642 3528 webserver.cc:460] Webserver started at http://127.0.98.1:42683/ using document root <none> and password file <none>
I20260927 16:39:00.308167 3528 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:00.308247 3528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:00.308444 3528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:00.310257 3528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/data/instance:
uuid: "f512e4a26368483096ca9834fe0ee2b2"
format_stamp: "Formatted at 2026-09-27 16:39:00 on dist-test-slave-z40b"
server_key: "34204af31f02fb821430167166619e13"
server_key_iv: "51358fd688ba9b274c4b77756696b8d2"
server_key_version: "encryptionkey@0"
I20260927 16:39:00.310768 3528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f512e4a26368483096ca9834fe0ee2b2"
format_stamp: "Formatted at 2026-09-27 16:39:00 on dist-test-slave-z40b"
server_key: "34204af31f02fb821430167166619e13"
server_key_iv: "51358fd688ba9b274c4b77756696b8d2"
server_key_version: "encryptionkey@0"
I20260927 16:39:00.314177 3528 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:39:00.316298 3543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:00.317305 3528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:39:00.317431 3528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "f512e4a26368483096ca9834fe0ee2b2"
format_stamp: "Formatted at 2026-09-27 16:39:00 on dist-test-slave-z40b"
server_key: "34204af31f02fb821430167166619e13"
server_key_iv: "51358fd688ba9b274c4b77756696b8d2"
server_key_version: "encryptionkey@0"
I20260927 16:39:00.317523 3528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:00.339157 3528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:00.341661 3528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:00.341821 3528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:00.342314 3528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:00.343060 3528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:00.343122 3528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:00.343192 3528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:00.343263 3528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:00.351414 3528 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33501
I20260927 16:39:00.351459 3656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33501 every 8 connection(s)
I20260927 16:39:00.352145 3528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:39:00.355098 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 3528
I20260927 16:39:00.355233 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:39:00.355536 392 external_mini_cluster.cc:1428] Setting key 1e0a60d93528d1a83e1a3c5b4c4bb439
Sep 27 16:39:00 dist-test-slave-z40b krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:39:00.366402 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:00.353701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51537 (local address 127.0.98.62:45169)
0927 16:39:00.353824 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:00.353828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:00.354558 (+ 730us) server_negotiation.cc:407] Connection header received
0927 16:39:00.356158 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:00.356162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:00.356211 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:00.356290 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:00.357526 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:00.358399 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:00.360165 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:00.360362 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:00.360457 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:00.362922 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:00.362927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:00.362930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:00.362955 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:00.364529 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:00.364986 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:00.364990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:00.364992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:00.365073 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:00.365420 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:00.365423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:00.365427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:00.365646 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:00.365725 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:00.366129 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 16:39:00.366275 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:39:00.367126 3659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:00.354027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45169 (local address 127.0.98.1:51537)
0927 16:39:00.354424 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:00.354461 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:00.355938 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:00.356463 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:00.356470 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:00.356936 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:00.357391 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:00.357403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:00.358533 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:00.358538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:00.360045 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:00.360058 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:00.360519 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:00.360526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:00.360743 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:00.361316 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:00.361334 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:00.362754 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:00.364637 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:00.364641 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:00.364651 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:00.364881 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:00.365162 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:00.365163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:00.365165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:00.365330 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:00.365730 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:00.365734 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:00.365942 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 16:39:00.366085 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 16:39:00.366334 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 16:39:00.368296 3657 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45169
I20260927 16:39:00.368546 3657 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:00.368993 3657 heartbeater.cc:511] Master 127.0.98.62:45169 requested a full tablet report, sending...
I20260927 16:39:00.370304 3466 ts_manager.cc:194] Registered new tserver with Master: f512e4a26368483096ca9834fe0ee2b2 (127.0.98.1:33501)
I20260927 16:39:00.371393 3466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:39:00.382604 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45169
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:00.483068 3662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:00.483343 3662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:00.483436 3662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:00.487445 3662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:00.487512 3662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:00.487581 3662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:39:00.492343 3662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:00.493212 3662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:00.494105 3662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:00.502663 3668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:00.502737 3670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:00.502774 3662 server_base.cc:1034] running on GCE node
W20260927 16:39:00.502941 3667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:00.503355 3662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:00.503934 3662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:00.505100 3662 hybrid_clock.cc:648] HybridClock initialized: now 1790527140505072 us; error 38 us; skew 500 ppm
Sep 27 16:39:00 dist-test-slave-z40b krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:00.507668 3662 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:39:00.508437 3662 webserver.cc:460] Webserver started at http://127.0.98.2:40179/ using document root <none> and password file <none>
I20260927 16:39:00.508948 3662 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:00.509032 3662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:00.509230 3662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:00.511062 3662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/data/instance:
uuid: "1bd758566f9b4b4b8ec88bbf6182622c"
format_stamp: "Formatted at 2026-09-27 16:39:00 on dist-test-slave-z40b"
server_key: "aab508e0519c3d9e1b5537219aff8923"
server_key_iv: "d1769a61a7fe1954361d5de1881be600"
server_key_version: "encryptionkey@0"
I20260927 16:39:00.511775 3662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1bd758566f9b4b4b8ec88bbf6182622c"
format_stamp: "Formatted at 2026-09-27 16:39:00 on dist-test-slave-z40b"
server_key: "aab508e0519c3d9e1b5537219aff8923"
server_key_iv: "d1769a61a7fe1954361d5de1881be600"
server_key_version: "encryptionkey@0"
I20260927 16:39:00.515481 3662 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:39:00.517700 3677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:00.518754 3662 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:00.518882 3662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "1bd758566f9b4b4b8ec88bbf6182622c"
format_stamp: "Formatted at 2026-09-27 16:39:00 on dist-test-slave-z40b"
server_key: "aab508e0519c3d9e1b5537219aff8923"
server_key_iv: "d1769a61a7fe1954361d5de1881be600"
server_key_version: "encryptionkey@0"
I20260927 16:39:00.518983 3662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:00.537111 3662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:00.539629 3662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:00.539798 3662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:00.540297 3662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:00.541033 3662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:00.541109 3662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:00.541165 3662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:00.541208 3662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:00.549096 3662 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41099
I20260927 16:39:00.549119 3790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41099 every 8 connection(s)
I20260927 16:39:00.549854 3662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:39:00.557801 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 3662
I20260927 16:39:00.557907 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:39:00.558147 392 external_mini_cluster.cc:1428] Setting key 809f22ca7bb617b4317f1d0bb0d5a309
Sep 27 16:39:00 dist-test-slave-z40b krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:39:00.562354 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:00.551500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60869 (local address 127.0.98.62:45169)
0927 16:39:00.551625 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:00.551629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:00.552386 (+ 757us) server_negotiation.cc:407] Connection header received
0927 16:39:00.553557 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:00.553560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:00.553612 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:00.553704 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:00.554704 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:00.555179 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:00.556208 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:00.556411 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:00.556514 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:00.558778 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:00.558781 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:00.558783 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:00.558828 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:00.560532 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:00.560965 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:00.560971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:00.560975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:00.561063 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:00.561442 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:00.561444 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:00.561445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:00.561589 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:00.561659 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:00.562169 (+ 510us) server_negotiation.cc:299] Negotiation successful
0927 16:39:00.562251 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:39:00.563110 3793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:00.551843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45169 (local address 127.0.98.2:60869)
0927 16:39:00.552243 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:00.552278 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:00.553373 (+ 1095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:00.553851 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:00.553857 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:00.554160 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:00.554551 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:00.554579 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:00.555326 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:00.555329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:00.556089 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:00.556098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:00.556550 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:00.556554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:00.556700 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:00.557183 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:00.557196 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:00.558637 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:00.560643 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:00.560647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:00.560655 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:00.560868 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:00.561172 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:00.561176 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:00.561179 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:00.561345 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:00.561677 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:00.561684 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:00.561946 (+ 262us) client_negotiation.cc:769] Sending connection context
0927 16:39:00.562144 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 16:39:00.562401 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":103,"threads_started":1}
I20260927 16:39:00.564067 3791 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45169
I20260927 16:39:00.564280 3791 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:00.564674 3791 heartbeater.cc:511] Master 127.0.98.62:45169 requested a full tablet report, sending...
I20260927 16:39:00.565603 3466 ts_manager.cc:194] Registered new tserver with Master: 1bd758566f9b4b4b8ec88bbf6182622c (127.0.98.2:41099)
I20260927 16:39:00.566123 3466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60869
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:39:00.582240 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45169
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:00.680606 3796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:00.680868 3796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:00.680963 3796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:00.684866 3796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:00.684949 3796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:00.685026 3796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:39:00.690121 3796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:00.690965 3796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:00.691895 3796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:00.700591 3804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:00.700608 3802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:00.700599 3801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:00.700842 3796 server_base.cc:1034] running on GCE node
I20260927 16:39:00.701212 3796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:00.701839 3796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:00.702988 3796 hybrid_clock.cc:648] HybridClock initialized: now 1790527140702959 us; error 38 us; skew 500 ppm
Sep 27 16:39:00 dist-test-slave-z40b krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:00.705500 3796 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:39:00.706243 3796 webserver.cc:460] Webserver started at http://127.0.98.3:40373/ using document root <none> and password file <none>
I20260927 16:39:00.706717 3796 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:00.706769 3796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:00.706910 3796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:00.708745 3796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/data/instance:
uuid: "2f21461e0fb54e60bd5162633d418b6d"
format_stamp: "Formatted at 2026-09-27 16:39:00 on dist-test-slave-z40b"
server_key: "8a74ee4e157f0216f95679173af36d14"
server_key_iv: "99a7a01625adf5a44c2d68a304413c60"
server_key_version: "encryptionkey@0"
I20260927 16:39:00.709231 3796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2f21461e0fb54e60bd5162633d418b6d"
format_stamp: "Formatted at 2026-09-27 16:39:00 on dist-test-slave-z40b"
server_key: "8a74ee4e157f0216f95679173af36d14"
server_key_iv: "99a7a01625adf5a44c2d68a304413c60"
server_key_version: "encryptionkey@0"
I20260927 16:39:00.712523 3796 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 16:39:00.714661 3811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:00.715783 3796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:39:00.715888 3796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "2f21461e0fb54e60bd5162633d418b6d"
format_stamp: "Formatted at 2026-09-27 16:39:00 on dist-test-slave-z40b"
server_key: "8a74ee4e157f0216f95679173af36d14"
server_key_iv: "99a7a01625adf5a44c2d68a304413c60"
server_key_version: "encryptionkey@0"
I20260927 16:39:00.715991 3796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:00.731535 3796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:00.734207 3796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:00.734372 3796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:00.734848 3796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:00.735605 3796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:00.735649 3796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:00.735706 3796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:00.735744 3796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:00.743712 3796 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41959
I20260927 16:39:00.743753 3924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41959 every 8 connection(s)
I20260927 16:39:00.744447 3796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:39:00.747104 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 3796
I20260927 16:39:00.747242 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790527121515808-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:39:00.747570 392 external_mini_cluster.cc:1428] Setting key a05ec4643f55283cd37c533d10d9473e
Sep 27 16:39:00 dist-test-slave-z40b krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:39:00.756282 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:00.745980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44027 (local address 127.0.98.62:45169)
0927 16:39:00.746113 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:00.746118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:00.746914 (+ 796us) server_negotiation.cc:407] Connection header received
0927 16:39:00.748470 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:00.748473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:00.748526 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:00.748626 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:00.749657 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:00.750177 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:00.751171 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:00.751419 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:00.751517 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:00.753578 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:00.753581 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:00.753582 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:00.753604 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:00.754703 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:00.755103 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:00.755106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:00.755107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:00.755173 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:00.755498 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:00.755500 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:00.755501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:00.755654 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:00.755725 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:00.756087 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 16:39:00.756173 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:39:00.756958 3927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:00.746361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45169 (local address 127.0.98.3:44027)
0927 16:39:00.746782 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:00.746819 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:00.748283 (+ 1464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:00.748793 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:00.748799 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:00.749120 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:00.749529 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:00.749540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:00.750297 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:00.750300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:00.751062 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:00.751069 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:00.751543 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:00.751546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:00.751713 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:00.752149 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:00.752163 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:00.753446 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:00.754795 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:00.754799 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:00.754808 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:00.755027 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:00.755297 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:00.755299 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:00.755301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:00.755426 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:00.755731 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:00.755735 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:00.755914 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 16:39:00.756048 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 16:39:00.756242 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":102,"threads_started":1}
I20260927 16:39:00.757930 3925 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45169
I20260927 16:39:00.758149 3925 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:00.758549 3925 heartbeater.cc:511] Master 127.0.98.62:45169 requested a full tablet report, sending...
I20260927 16:39:00.759457 3466 ts_manager.cc:194] Registered new tserver with Master: 2f21461e0fb54e60bd5162633d418b6d (127.0.98.3:41959)
I20260927 16:39:00.760169 3466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44027
I20260927 16:39:00.761090 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 16:39:00 dist-test-slave-z40b krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527140, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 16:39:00 dist-test-slave-z40b krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527140, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:39:00.784264 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:00.776991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51818 (local address 127.0.98.62:45169)
0927 16:39:00.777122 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:00.777126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:00.777282 (+ 156us) server_negotiation.cc:407] Connection header received
0927 16:39:00.777369 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:00.777371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:00.777407 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:00.777468 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:00.778053 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:00.778501 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:00.779343 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:00.779541 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:00.779614 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:00.781680 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:00.781683 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:00.781685 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:00.781705 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:00.782817 (+ 1112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:00.783176 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:00.783178 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:00.783179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:00.783291 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:00.783521 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:00.783523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:00.783524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:00.783717 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:00.783813 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:00.783985 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 16:39:00.784066 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 16:39:00.784587 3466 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51818
I20260927 16:39:00.785719 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 3528
I20260927 16:39:00.792280 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 3662
I20260927 16:39:00.797340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 3796
I20260927 16:39:00.802276 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 3434
2026-09-27T16:39:00Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2023 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:39:00 dist-test-slave-z40b krb5kdc[3937](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:39:00 dist-test-slave-z40b krb5kdc[3937](info): set up 2 sockets
Sep 27 16:39:00 dist-test-slave-z40b krb5kdc[3937](info): commencing operation
krb5kdc: starting...
W20260927 16:39:01.811079 392 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:39:01 dist-test-slave-z40b krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:39:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:39:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:39:01.922549 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46663
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:02.023231 3950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:02.023494 3950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:02.023576 3950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:02.027700 3950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:39:02.027770 3950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:02.027802 3950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:02.027850 3950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:39:02.027874 3950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:39:02.033104 3950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:02.033958 3950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:02.034808 3950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:02.042240 3956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:02.042402 3950 server_base.cc:1034] running on GCE node
W20260927 16:39:02.042240 3955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:02.042263 3958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:02.043097 3950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:02.044034 3950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:02.045194 3950 hybrid_clock.cc:648] HybridClock initialized: now 1790527142045174 us; error 32 us; skew 500 ppm
Sep 27 16:39:02 dist-test-slave-z40b krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:02.047737 3950 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:39:02.048530 3950 webserver.cc:460] Webserver started at http://127.0.98.62:40227/ using document root <none> and password file <none>
I20260927 16:39:02.049037 3950 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:02.049117 3950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:02.049320 3950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:02.051124 3950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "62d4157a623345999f2b776f72142ab2"
format_stamp: "Formatted at 2026-09-27 16:39:02 on dist-test-slave-z40b"
server_key: "b939ad3c5ffba108ac08a4c1b94194c4"
server_key_iv: "0b08d0ce46f90defcf253b8733bb41cd"
server_key_version: "encryptionkey@0"
I20260927 16:39:02.051677 3950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "62d4157a623345999f2b776f72142ab2"
format_stamp: "Formatted at 2026-09-27 16:39:02 on dist-test-slave-z40b"
server_key: "b939ad3c5ffba108ac08a4c1b94194c4"
server_key_iv: "0b08d0ce46f90defcf253b8733bb41cd"
server_key_version: "encryptionkey@0"
I20260927 16:39:02.054867 3950 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:39:02.057106 3965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:02.058167 3950 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:02.058292 3950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "62d4157a623345999f2b776f72142ab2"
format_stamp: "Formatted at 2026-09-27 16:39:02 on dist-test-slave-z40b"
server_key: "b939ad3c5ffba108ac08a4c1b94194c4"
server_key_iv: "0b08d0ce46f90defcf253b8733bb41cd"
server_key_version: "encryptionkey@0"
I20260927 16:39:02.058390 3950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:02.071382 3950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:02.073609 3950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:02.073779 3950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:02.080317 3950 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46663
I20260927 16:39:02.080332 4017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46663 every 8 connection(s)
I20260927 16:39:02.081152 3950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:39:02.083684 4018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:02.087553 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 3950
I20260927 16:39:02.087680 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:39:02.087934 392 external_mini_cluster.cc:1428] Setting key 9313871675d18b2286228eeb936bbeee
I20260927 16:39:02.089107 4018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62d4157a623345999f2b776f72142ab2: Bootstrap starting.
I20260927 16:39:02.091358 4018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62d4157a623345999f2b776f72142ab2: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:02.092096 4018 log.cc:824] T 00000000000000000000000000000000 P 62d4157a623345999f2b776f72142ab2: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:02.093729 4018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62d4157a623345999f2b776f72142ab2: No bootstrap required, opened a new log
Sep 27 16:39:02 dist-test-slave-z40b krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:39:02.097736 4018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62d4157a623345999f2b776f72142ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62d4157a623345999f2b776f72142ab2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46663 } }
I20260927 16:39:02.097925 4018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62d4157a623345999f2b776f72142ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:02.097985 4018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62d4157a623345999f2b776f72142ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62d4157a623345999f2b776f72142ab2, State: Initialized, Role: FOLLOWER
I20260927 16:39:02.098366 4018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62d4157a623345999f2b776f72142ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62d4157a623345999f2b776f72142ab2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46663 } }
I20260927 16:39:02.098518 4018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62d4157a623345999f2b776f72142ab2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:39:02.098604 4018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62d4157a623345999f2b776f72142ab2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:39:02.098702 4018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62d4157a623345999f2b776f72142ab2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:02.099637 4018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62d4157a623345999f2b776f72142ab2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62d4157a623345999f2b776f72142ab2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46663 } }
I20260927 16:39:02.099992 4018 leader_election.cc:302] T 00000000000000000000000000000000 P 62d4157a623345999f2b776f72142ab2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62d4157a623345999f2b776f72142ab2; no voters:
I20260927 16:39:02.100325 4018 leader_election.cc:288] T 00000000000000000000000000000000 P 62d4157a623345999f2b776f72142ab2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:39:02.100399 4023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62d4157a623345999f2b776f72142ab2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:39:02.100610 4023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62d4157a623345999f2b776f72142ab2 [term 1 LEADER]: Becoming Leader. State: Replica: 62d4157a623345999f2b776f72142ab2, State: Running, Role: LEADER
I20260927 16:39:02.100890 4023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62d4157a623345999f2b776f72142ab2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62d4157a623345999f2b776f72142ab2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46663 } }
I20260927 16:39:02.101327 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:02.089119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56392 (local address 127.0.98.62:46663)
0927 16:39:02.089502 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:02.089524 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:02.089556 (+ 32us) server_negotiation.cc:407] Connection header received
0927 16:39:02.090157 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:02.090183 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:02.090462 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:02.090705 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:02.091608 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:02.092564 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:02.093399 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:02.093858 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:02.094025 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:02.096682 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:02.096687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:02.096698 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:02.096721 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:02.098370 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:02.098846 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:02.098851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:02.098861 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:02.098969 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:02.099295 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:02.099298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:02.099299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:02.099630 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:02.099811 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:02.100044 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 16:39:02.100333 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":116,"threads_started":1}
I20260927 16:39:02.101558 4018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62d4157a623345999f2b776f72142ab2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:39:02.101963 4023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62d4157a623345999f2b776f72142ab2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62d4157a623345999f2b776f72142ab2. Latest consensus state: current_term: 1 leader_uuid: "62d4157a623345999f2b776f72142ab2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62d4157a623345999f2b776f72142ab2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46663 } } }
I20260927 16:39:02.102061 4023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62d4157a623345999f2b776f72142ab2 [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:02.102342 4024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62d4157a623345999f2b776f72142ab2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62d4157a623345999f2b776f72142ab2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62d4157a623345999f2b776f72142ab2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46663 } } }
I20260927 16:39:02.102448 4024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62d4157a623345999f2b776f72142ab2 [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:02.102577 4027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:39:02.105474 4027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:39:02.110733 4027 catalog_manager.cc:1347] Generated new cluster ID: 79dcea3fea3f437480e67891ceb9b1c3
I20260927 16:39:02.110800 4027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:39:02.134007 4027 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:39:02.135048 4027 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:39:02.139071 4027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62d4157a623345999f2b776f72142ab2: Generated new TSK 0
I20260927 16:39:02.139757 4027 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:39:02.167133 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46663
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:02.271770 4044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:02.272014 4044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:02.272084 4044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:02.275983 4044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:02.276028 4044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:02.276116 4044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:39:02.280912 4044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:02.281832 4044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:02.282748 4044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:02.291447 4049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:02.291448 4052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:02.291607 4044 server_base.cc:1034] running on GCE node
W20260927 16:39:02.291486 4050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:02.292054 4044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:02.292615 4044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:02.293754 4044 hybrid_clock.cc:648] HybridClock initialized: now 1790527142293740 us; error 24 us; skew 500 ppm
Sep 27 16:39:02 dist-test-slave-z40b krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:02.296299 4044 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:39:02.297019 4044 webserver.cc:460] Webserver started at http://127.0.98.1:44501/ using document root <none> and password file <none>
I20260927 16:39:02.297446 4044 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:02.297504 4044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:02.297640 4044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:02.299405 4044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-0/data/instance:
uuid: "dba240db0be1447dbae0853b9c6a1081"
format_stamp: "Formatted at 2026-09-27 16:39:02 on dist-test-slave-z40b"
server_key: "a77077e700449364ac3f1f29688f5628"
server_key_iv: "f547eccba29f9856d9a76be146dc0628"
server_key_version: "encryptionkey@0"
I20260927 16:39:02.299885 4044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dba240db0be1447dbae0853b9c6a1081"
format_stamp: "Formatted at 2026-09-27 16:39:02 on dist-test-slave-z40b"
server_key: "a77077e700449364ac3f1f29688f5628"
server_key_iv: "f547eccba29f9856d9a76be146dc0628"
server_key_version: "encryptionkey@0"
I20260927 16:39:02.303321 4044 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:39:02.305438 4059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:02.306536 4044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:02.306661 4044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "dba240db0be1447dbae0853b9c6a1081"
format_stamp: "Formatted at 2026-09-27 16:39:02 on dist-test-slave-z40b"
server_key: "a77077e700449364ac3f1f29688f5628"
server_key_iv: "f547eccba29f9856d9a76be146dc0628"
server_key_version: "encryptionkey@0"
I20260927 16:39:02.306758 4044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:02.320266 4044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:02.323120 4044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:02.323382 4044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:02.323889 4044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:02.324647 4044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:02.324712 4044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:02.324781 4044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:02.324833 4044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:02.332944 4044 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37409
I20260927 16:39:02.332978 4172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37409 every 8 connection(s)
I20260927 16:39:02.333740 4044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 16:39:02 dist-test-slave-z40b krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:39:02.342058 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 4044
I20260927 16:39:02.342171 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:39:02.342435 392 external_mini_cluster.cc:1428] Setting key 8d5a5dcd2a6eb94e8615350342a57c02
I20260927 16:39:02.346035 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:02.335305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35751 (local address 127.0.98.62:46663)
0927 16:39:02.335430 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:02.335435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:02.336209 (+ 774us) server_negotiation.cc:407] Connection header received
0927 16:39:02.337347 (+ 1138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:02.337349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:02.337390 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:02.337457 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:02.338494 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:02.339161 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:02.340248 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:02.340454 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:02.340526 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:02.342641 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:02.342649 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:02.342651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:02.342669 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:02.344179 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:02.344640 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:02.344645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:02.344647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:02.344721 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:02.345025 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:02.345030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:02.345032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:02.345194 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:02.345290 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:02.345750 (+ 460us) server_negotiation.cc:299] Negotiation successful
0927 16:39:02.345864 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 16:39:02.346738 4175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:02.335663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46663 (local address 127.0.98.1:35751)
0927 16:39:02.336059 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:02.336097 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:02.337179 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:02.337598 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:02.337604 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:02.337939 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:02.338345 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:02.338356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:02.339306 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:02.339308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:02.340129 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:02.340137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:02.340564 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:02.340567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:02.340688 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:02.341129 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:02.341143 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:02.342484 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:02.344286 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:02.344291 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:02.344304 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:02.344544 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:02.344803 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:02.344807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:02.344809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:02.344935 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:02.345288 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:02.345294 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:02.345540 (+ 246us) client_negotiation.cc:769] Sending connection context
0927 16:39:02.345711 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 16:39:02.345994 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":106,"threads_started":1}
I20260927 16:39:02.347811 4173 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46663
I20260927 16:39:02.348038 4173 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:02.348574 4173 heartbeater.cc:511] Master 127.0.98.62:46663 requested a full tablet report, sending...
I20260927 16:39:02.350219 3982 ts_manager.cc:194] Registered new tserver with Master: dba240db0be1447dbae0853b9c6a1081 (127.0.98.1:37409)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:39:02.351840 3982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:39:02.367828 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46663
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:02.470669 4178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:02.470870 4178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:02.470906 4178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:02.474325 4178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:02.474366 4178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:02.474434 4178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:39:02.478945 4178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:02.479820 4178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:02.480664 4178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:02.489244 4184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:02.489290 4186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:02.489284 4183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:02.489398 4178 server_base.cc:1034] running on GCE node
I20260927 16:39:02.489825 4178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:02.490407 4178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:02.491578 4178 hybrid_clock.cc:648] HybridClock initialized: now 1790527142491568 us; error 27 us; skew 500 ppm
Sep 27 16:39:02 dist-test-slave-z40b krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:02.493968 4178 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:39:02.494675 4178 webserver.cc:460] Webserver started at http://127.0.98.2:36713/ using document root <none> and password file <none>
I20260927 16:39:02.495159 4178 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:02.495258 4178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:02.495438 4178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:02.497094 4178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-1/data/instance:
uuid: "243a6cfd449f454db1b0cffa43f7b9d6"
format_stamp: "Formatted at 2026-09-27 16:39:02 on dist-test-slave-z40b"
server_key: "fa8933044f68bc11ecc993c2a5b3fd34"
server_key_iv: "ace55d471bbe9d0818c59342664e3e1f"
server_key_version: "encryptionkey@0"
I20260927 16:39:02.497531 4178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-1/wal/instance:
uuid: "243a6cfd449f454db1b0cffa43f7b9d6"
format_stamp: "Formatted at 2026-09-27 16:39:02 on dist-test-slave-z40b"
server_key: "fa8933044f68bc11ecc993c2a5b3fd34"
server_key_iv: "ace55d471bbe9d0818c59342664e3e1f"
server_key_version: "encryptionkey@0"
I20260927 16:39:02.500754 4178 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:39:02.503090 4193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:02.504225 4178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:39:02.504308 4178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "243a6cfd449f454db1b0cffa43f7b9d6"
format_stamp: "Formatted at 2026-09-27 16:39:02 on dist-test-slave-z40b"
server_key: "fa8933044f68bc11ecc993c2a5b3fd34"
server_key_iv: "ace55d471bbe9d0818c59342664e3e1f"
server_key_version: "encryptionkey@0"
I20260927 16:39:02.504380 4178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:02.531643 4178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:02.534090 4178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:02.534225 4178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:02.534678 4178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:02.535425 4178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:02.535476 4178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:02.535513 4178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:02.535533 4178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:02.543466 4178 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33571
I20260927 16:39:02.543515 4306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33571 every 8 connection(s)
I20260927 16:39:02.544253 4178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 16:39:02 dist-test-slave-z40b krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:39:02.553120 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 4178
I20260927 16:39:02.553246 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:39:02.553498 392 external_mini_cluster.cc:1428] Setting key d0a3192e6542963bc6e3b9e88f99d71e
I20260927 16:39:02.557018 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:02.545816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52873 (local address 127.0.98.62:46663)
0927 16:39:02.545937 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:02.545949 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:02.546650 (+ 701us) server_negotiation.cc:407] Connection header received
0927 16:39:02.547893 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:02.547896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:02.547957 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:02.548040 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:02.549157 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:02.549669 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:02.550759 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:02.550974 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:02.551072 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:02.553379 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:02.553382 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:02.553384 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:02.553403 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:02.554883 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:02.555458 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:02.555461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:02.555463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:02.555545 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:02.555879 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:02.555880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:02.555881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:02.556027 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:02.556093 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:02.556695 (+ 602us) server_negotiation.cc:299] Negotiation successful
0927 16:39:02.556814 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:39:02.557662 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:02.546103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46663 (local address 127.0.98.2:52873)
0927 16:39:02.546493 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:02.546542 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:02.547704 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:02.548186 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:02.548192 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:02.548493 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:02.549016 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:02.549029 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:02.549783 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:02.549788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:02.550641 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:02.550650 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:02.551124 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:02.551128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:02.551322 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:02.551887 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:02.551902 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:02.553224 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:02.554983 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:02.554988 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:02.554999 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:02.555367 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:02.555625 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:02.555630 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:02.555634 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:02.555795 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:02.556116 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:02.556124 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:02.556403 (+ 279us) client_negotiation.cc:769] Sending connection context
0927 16:39:02.556586 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 16:39:02.556859 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":94,"threads_started":1}
I20260927 16:39:02.558846 4307 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46663
I20260927 16:39:02.559051 4307 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:02.559522 4307 heartbeater.cc:511] Master 127.0.98.62:46663 requested a full tablet report, sending...
I20260927 16:39:02.560456 3982 ts_manager.cc:194] Registered new tserver with Master: 243a6cfd449f454db1b0cffa43f7b9d6 (127.0.98.2:33571)
I20260927 16:39:02.560961 3982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52873
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:39:02.578790 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46663
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:02.680358 4312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:02.680601 4312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:02.680639 4312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:02.684440 4312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:02.684482 4312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:02.684531 4312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:39:02.689263 4312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:02.690137 4312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:02.691046 4312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:02.699807 4317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:02.699927 4312 server_base.cc:1034] running on GCE node
W20260927 16:39:02.699875 4320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:02.699805 4318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:02.700470 4312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:02.701020 4312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:02.702191 4312 hybrid_clock.cc:648] HybridClock initialized: now 1790527142702162 us; error 40 us; skew 500 ppm
Sep 27 16:39:02 dist-test-slave-z40b krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:02.704792 4312 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:39:02.705555 4312 webserver.cc:460] Webserver started at http://127.0.98.3:38787/ using document root <none> and password file <none>
I20260927 16:39:02.706064 4312 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:02.706146 4312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:02.706352 4312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:02.708351 4312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-2/data/instance:
uuid: "01ef5012bec841d19804a3b7e1a82057"
format_stamp: "Formatted at 2026-09-27 16:39:02 on dist-test-slave-z40b"
server_key: "dbb45467a0d79cb9e9edbc470c479dc8"
server_key_iv: "4db7b31d6f96bcc417444052e1e3095f"
server_key_version: "encryptionkey@0"
I20260927 16:39:02.708868 4312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-2/wal/instance:
uuid: "01ef5012bec841d19804a3b7e1a82057"
format_stamp: "Formatted at 2026-09-27 16:39:02 on dist-test-slave-z40b"
server_key: "dbb45467a0d79cb9e9edbc470c479dc8"
server_key_iv: "4db7b31d6f96bcc417444052e1e3095f"
server_key_version: "encryptionkey@0"
I20260927 16:39:02.712348 4312 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:39:02.714437 4327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:02.715592 4312 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:39:02.715710 4312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "01ef5012bec841d19804a3b7e1a82057"
format_stamp: "Formatted at 2026-09-27 16:39:02 on dist-test-slave-z40b"
server_key: "dbb45467a0d79cb9e9edbc470c479dc8"
server_key_iv: "4db7b31d6f96bcc417444052e1e3095f"
server_key_version: "encryptionkey@0"
I20260927 16:39:02.715799 4312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:02.744704 4312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:02.747517 4312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:02.747700 4312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:02.748225 4312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:02.748955 4312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:02.749044 4312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:02.749127 4312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:02.749172 4312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:02.757161 4312 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37547
I20260927 16:39:02.757187 4440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37547 every 8 connection(s)
I20260927 16:39:02.757969 4312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:39:02.764006 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 4312
I20260927 16:39:02.764117 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790527121515808-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:39:02.764370 392 external_mini_cluster.cc:1428] Setting key f19e7e4d8afdb693c3c7966d266db7e2
Sep 27 16:39:02 dist-test-slave-z40b krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:39:02.770550 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:02.759660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39811 (local address 127.0.98.62:46663)
0927 16:39:02.759777 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:02.759781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:02.760477 (+ 696us) server_negotiation.cc:407] Connection header received
0927 16:39:02.761818 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:02.761822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:02.761896 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:02.761988 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:02.763101 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:02.763673 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:02.764926 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:02.765149 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:02.765250 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:02.767499 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:02.767503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:02.767506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:02.767535 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:02.768683 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:02.769162 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:02.769167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:02.769169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:02.769245 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:02.769552 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:02.769556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:02.769558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:02.769722 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:02.769822 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:02.770278 (+ 456us) server_negotiation.cc:299] Negotiation successful
0927 16:39:02.770385 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 16:39:02.771203 4443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:02.759953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46663 (local address 127.0.98.3:39811)
0927 16:39:02.760344 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:02.760381 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:02.761607 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:02.762180 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:02.762190 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:02.762542 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:02.762951 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:02.762963 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:02.763788 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:02.763796 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:02.764802 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:02.764813 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:02.765282 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:02.765286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:02.765454 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:02.765965 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:02.765980 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:02.767349 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:02.768789 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:02.768794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:02.768807 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:02.769063 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:02.769330 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:02.769335 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:02.769337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:02.769461 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:02.769818 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:02.769824 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:02.770047 (+ 223us) client_negotiation.cc:769] Sending connection context
0927 16:39:02.770215 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 16:39:02.770496 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":94,"threads_started":1}
I20260927 16:39:02.772119 4441 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46663
I20260927 16:39:02.772291 4441 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:02.772672 4441 heartbeater.cc:511] Master 127.0.98.62:46663 requested a full tablet report, sending...
I20260927 16:39:02.773483 3982 ts_manager.cc:194] Registered new tserver with Master: 01ef5012bec841d19804a3b7e1a82057 (127.0.98.3:37547)
I20260927 16:39:02.773950 3982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39811
I20260927 16:39:02.777498 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 16:39:02 dist-test-slave-z40b krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 16:39:02 dist-test-slave-z40b krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 16:39:02.801992 4448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:02.792461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36776 (local address 127.0.98.1:37409)
0927 16:39:02.792763 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:02.792776 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:02.792797 (+ 21us) server_negotiation.cc:407] Connection header received
0927 16:39:02.792837 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:02.792841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:02.792982 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:02.793092 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:02.793681 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:02.794582 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:02.795734 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:02.796266 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:02.796371 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:02.798805 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:02.798813 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:02.798825 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:02.798852 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:02.800250 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:02.800715 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:02.800725 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:02.800727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:02.800799 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:02.801061 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:02.801066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:02.801067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:02.801280 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:02.801542 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:02.801680 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 16:39:02.801822 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":99,"threads_started":1}
Sep 27 16:39:02 dist-test-slave-z40b krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:39:02.812100 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:02.804355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56396 (local address 127.0.98.62:46663)
0927 16:39:02.804499 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:02.804503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:02.804571 (+ 68us) server_negotiation.cc:407] Connection header received
0927 16:39:02.804697 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:02.804700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:02.804738 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:02.804820 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:02.805353 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:02.805860 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:02.806615 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:02.806928 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:02.807027 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:02.809327 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:02.809330 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:02.809332 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:02.809352 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:02.810574 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:02.810953 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:02.810955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:02.810957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:02.811045 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:02.811320 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:02.811322 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:02.811325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:02.811580 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:02.811691 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:02.811842 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 16:39:02.811920 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 16:39:02.812434 3982 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56396
I20260927 16:39:02.812877 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 4044
I20260927 16:39:02.819264 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 4178
I20260927 16:39:02.824817 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 4312
I20260927 16:39:02.830499 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 3950
2026-09-27T16:39:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2028 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:39:02 dist-test-slave-z40b krb5kdc[4453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:39:02 dist-test-slave-z40b krb5kdc[4453](info): set up 2 sockets
Sep 27 16:39:02 dist-test-slave-z40b krb5kdc[4453](info): commencing operation
krb5kdc: starting...
W20260927 16:39:03.835040 392 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:39:03 dist-test-slave-z40b krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:39:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:39:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:39:03.945736 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37873
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:04.045953 4466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:04.046159 4466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:04.046196 4466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:04.049998 4466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:39:04.050041 4466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:04.050060 4466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:04.050076 4466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:39:04.050091 4466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:39:04.054962 4466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:04.055814 4466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:04.056649 4466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:04.063835 4472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:04.063827 4471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:04.063911 4466 server_base.cc:1034] running on GCE node
W20260927 16:39:04.063880 4474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:04.064662 4466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:04.065654 4466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:04.066834 4466 hybrid_clock.cc:648] HybridClock initialized: now 1790527144066819 us; error 28 us; skew 500 ppm
Sep 27 16:39:04 dist-test-slave-z40b krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:04.075443 4466 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:39:04.076083 4466 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:04.076172 4466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:04.076387 4466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:04.078195 4466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "10abab21879e48938901c8eabbeae6d2"
format_stamp: "Formatted at 2026-09-27 16:39:04 on dist-test-slave-z40b"
server_key: "b5c4bccd8dbd9db86b8f2add46cf520d"
server_key_iv: "6af4081e0d2b18b9cf938f45a0c60af9"
server_key_version: "encryptionkey@0"
I20260927 16:39:04.078697 4466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "10abab21879e48938901c8eabbeae6d2"
format_stamp: "Formatted at 2026-09-27 16:39:04 on dist-test-slave-z40b"
server_key: "b5c4bccd8dbd9db86b8f2add46cf520d"
server_key_iv: "6af4081e0d2b18b9cf938f45a0c60af9"
server_key_version: "encryptionkey@0"
I20260927 16:39:04.082103 4466 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:39:04.084293 4480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:04.085327 4466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:39:04.085492 4466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "10abab21879e48938901c8eabbeae6d2"
format_stamp: "Formatted at 2026-09-27 16:39:04 on dist-test-slave-z40b"
server_key: "b5c4bccd8dbd9db86b8f2add46cf520d"
server_key_iv: "6af4081e0d2b18b9cf938f45a0c60af9"
server_key_version: "encryptionkey@0"
I20260927 16:39:04.085605 4466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:04.102811 4466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:04.105355 4466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:04.105588 4466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:04.112258 4466 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37873
I20260927 16:39:04.112540 4532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37873 every 8 connection(s)
I20260927 16:39:04.112887 4466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:39:04.115466 4533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:04.120591 4533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10abab21879e48938901c8eabbeae6d2: Bootstrap starting.
I20260927 16:39:04.120754 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 4466
I20260927 16:39:04.120887 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:39:04.121205 392 external_mini_cluster.cc:1428] Setting key 9fee96e7a797b79241a500f76ce57827
I20260927 16:39:04.123186 4533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10abab21879e48938901c8eabbeae6d2: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:04.123945 4533 log.cc:824] T 00000000000000000000000000000000 P 10abab21879e48938901c8eabbeae6d2: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:04.125695 4533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10abab21879e48938901c8eabbeae6d2: No bootstrap required, opened a new log
I20260927 16:39:04.128955 4533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10abab21879e48938901c8eabbeae6d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10abab21879e48938901c8eabbeae6d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37873 } }
I20260927 16:39:04.129245 4533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10abab21879e48938901c8eabbeae6d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:04.129292 4533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10abab21879e48938901c8eabbeae6d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10abab21879e48938901c8eabbeae6d2, State: Initialized, Role: FOLLOWER
Sep 27 16:39:04 dist-test-slave-z40b krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:39:04.129751 4533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10abab21879e48938901c8eabbeae6d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10abab21879e48938901c8eabbeae6d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37873 } }
I20260927 16:39:04.129905 4533 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10abab21879e48938901c8eabbeae6d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:39:04.129977 4533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10abab21879e48938901c8eabbeae6d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:39:04.130092 4533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10abab21879e48938901c8eabbeae6d2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:04.131186 4533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10abab21879e48938901c8eabbeae6d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10abab21879e48938901c8eabbeae6d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37873 } }
I20260927 16:39:04.131626 4533 leader_election.cc:302] T 00000000000000000000000000000000 P 10abab21879e48938901c8eabbeae6d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10abab21879e48938901c8eabbeae6d2; no voters:
I20260927 16:39:04.131932 4533 leader_election.cc:288] T 00000000000000000000000000000000 P 10abab21879e48938901c8eabbeae6d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:39:04.132057 4538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10abab21879e48938901c8eabbeae6d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:39:04.132273 4538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10abab21879e48938901c8eabbeae6d2 [term 1 LEADER]: Becoming Leader. State: Replica: 10abab21879e48938901c8eabbeae6d2, State: Running, Role: LEADER
I20260927 16:39:04.132550 4538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10abab21879e48938901c8eabbeae6d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10abab21879e48938901c8eabbeae6d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37873 } }
I20260927 16:39:04.133206 4533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10abab21879e48938901c8eabbeae6d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:39:04.134013 4539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10abab21879e48938901c8eabbeae6d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10abab21879e48938901c8eabbeae6d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10abab21879e48938901c8eabbeae6d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37873 } } }
I20260927 16:39:04.134142 4539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10abab21879e48938901c8eabbeae6d2 [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:04.134433 4540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10abab21879e48938901c8eabbeae6d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10abab21879e48938901c8eabbeae6d2. Latest consensus state: current_term: 1 leader_uuid: "10abab21879e48938901c8eabbeae6d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10abab21879e48938901c8eabbeae6d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37873 } } }
I20260927 16:39:04.134509 4540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10abab21879e48938901c8eabbeae6d2 [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:04.135371 4547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:39:04.136332 4536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:04.122413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56838 (local address 127.0.98.62:37873)
0927 16:39:04.122834 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:04.122848 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:04.122882 (+ 34us) server_negotiation.cc:407] Connection header received
0927 16:39:04.123597 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:04.123614 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:04.123949 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:04.124317 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:04.125101 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.126104 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.126973 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.127507 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.127681 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:04.130296 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:04.130316 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:04.130331 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:04.130361 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:04.132401 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:04.132866 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:04.132872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:04.132877 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:04.133010 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:04.133359 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:04.133362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:04.133364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:04.133946 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:04.134915 (+ 969us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:04.135064 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 16:39:04.135365 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":296,"thread_start_us":126,"threads_started":1}
I20260927 16:39:04.138188 4547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:39:04.143388 4547 catalog_manager.cc:1347] Generated new cluster ID: 1e1f64bfff294a5c9a4303452b70e4e4
I20260927 16:39:04.143443 4547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:39:04.156070 4547 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:39:04.157191 4547 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:39:04.166498 4547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10abab21879e48938901c8eabbeae6d2: Generated new TSK 0
I20260927 16:39:04.167162 4547 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:39:04.200009 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37873
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:04.304628 4559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:04.304876 4559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:04.304943 4559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:04.308718 4559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:04.308785 4559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:04.308881 4559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:39:04.313791 4559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:04.314700 4559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:04.315696 4559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:04.324234 4565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:04.324239 4567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:04.324249 4559 server_base.cc:1034] running on GCE node
W20260927 16:39:04.324267 4564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:04.324922 4559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:04.325516 4559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:04.326704 4559 hybrid_clock.cc:648] HybridClock initialized: now 1790527144326697 us; error 28 us; skew 500 ppm
Sep 27 16:39:04 dist-test-slave-z40b krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:04.329427 4559 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:39:04.330091 4559 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:04.330179 4559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:04.330413 4559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:04.332280 4559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-0/data/instance:
uuid: "bef55520bc2c4ceb9ff04aa1f74cdbf1"
format_stamp: "Formatted at 2026-09-27 16:39:04 on dist-test-slave-z40b"
server_key: "5c80cbf7bdb31a7763c2d9b3dd900417"
server_key_iv: "f2cd574ae643ab93be1e0fa112f33043"
server_key_version: "encryptionkey@0"
I20260927 16:39:04.332861 4559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bef55520bc2c4ceb9ff04aa1f74cdbf1"
format_stamp: "Formatted at 2026-09-27 16:39:04 on dist-test-slave-z40b"
server_key: "5c80cbf7bdb31a7763c2d9b3dd900417"
server_key_iv: "f2cd574ae643ab93be1e0fa112f33043"
server_key_version: "encryptionkey@0"
I20260927 16:39:04.336294 4559 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:39:04.338428 4573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:04.339568 4559 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:39:04.339694 4559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "bef55520bc2c4ceb9ff04aa1f74cdbf1"
format_stamp: "Formatted at 2026-09-27 16:39:04 on dist-test-slave-z40b"
server_key: "5c80cbf7bdb31a7763c2d9b3dd900417"
server_key_iv: "f2cd574ae643ab93be1e0fa112f33043"
server_key_version: "encryptionkey@0"
I20260927 16:39:04.339794 4559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:04.355934 4559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:04.358755 4559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:04.358943 4559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:04.359490 4559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:04.360220 4559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:04.360291 4559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:04.360364 4559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:04.360410 4559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:04.368834 4559 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44389
I20260927 16:39:04.368848 4686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44389 every 8 connection(s)
I20260927 16:39:04.369403 4559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:39:04.375454 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 4559
I20260927 16:39:04.375597 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:39:04.375875 392 external_mini_cluster.cc:1428] Setting key 76aae1dd9799305d49e8f399f7ba2e3d
Sep 27 16:39:04 dist-test-slave-z40b krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:39:04.382030 4536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:04.370905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48951 (local address 127.0.98.62:37873)
0927 16:39:04.371024 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:04.371027 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:04.371753 (+ 726us) server_negotiation.cc:407] Connection header received
0927 16:39:04.372907 (+ 1154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:04.372910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:04.372962 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:04.373053 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:04.374055 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.374537 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.375688 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.376006 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.376118 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:04.378403 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:04.378409 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:04.378415 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:04.378449 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:04.380151 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:04.380617 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:04.380624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:04.380628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:04.380719 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:04.381026 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:04.381029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:04.381031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:04.381200 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:04.381301 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:04.381744 (+ 443us) server_negotiation.cc:299] Negotiation successful
0927 16:39:04.381857 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:39:04.382725 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:04.371169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37873 (local address 127.0.98.1:48951)
0927 16:39:04.371622 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:04.371657 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:04.372742 (+ 1085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:04.373193 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:04.373199 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:04.373533 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:04.373934 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:04.373945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.374635 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.374637 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:04.375541 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:04.375565 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.376166 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.376170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:04.376322 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:04.376840 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:04.376862 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:04.378256 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:04.380258 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:04.380262 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:04.380273 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:04.380523 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:04.380811 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:04.380813 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:04.380814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:04.380925 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:04.381310 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:04.381318 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:04.381526 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 16:39:04.381689 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 16:39:04.381962 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":96,"threads_started":1}
I20260927 16:39:04.383898 4687 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37873
I20260927 16:39:04.384150 4687 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:04.384640 4687 heartbeater.cc:511] Master 127.0.98.62:37873 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:39:04.386118 4497 ts_manager.cc:194] Registered new tserver with Master: bef55520bc2c4ceb9ff04aa1f74cdbf1 (127.0.98.1:44389)
I20260927 16:39:04.387138 4497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:39:04.402230 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37873
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:04.503309 4692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:04.503520 4692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:04.503566 4692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:04.507242 4692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:04.507282 4692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:04.507331 4692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:39:04.512029 4692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:04.512833 4692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:04.513669 4692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:04.522415 4698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:04.522475 4692 server_base.cc:1034] running on GCE node
W20260927 16:39:04.522418 4697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:04.522430 4700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:04.523080 4692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:04.523669 4692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:04.524838 4692 hybrid_clock.cc:648] HybridClock initialized: now 1790527144524826 us; error 34 us; skew 500 ppm
Sep 27 16:39:04 dist-test-slave-z40b krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:04.527485 4692 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:39:04.528141 4692 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:04.528223 4692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:04.528420 4692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:04.530200 4692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-1/data/instance:
uuid: "2dea7a3803074d469ce4260955a64b55"
format_stamp: "Formatted at 2026-09-27 16:39:04 on dist-test-slave-z40b"
server_key: "6ca76076a4da4a42c33ddc953b4e21b6"
server_key_iv: "2f7898458183605f773f03558badc0ea"
server_key_version: "encryptionkey@0"
I20260927 16:39:04.530714 4692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2dea7a3803074d469ce4260955a64b55"
format_stamp: "Formatted at 2026-09-27 16:39:04 on dist-test-slave-z40b"
server_key: "6ca76076a4da4a42c33ddc953b4e21b6"
server_key_iv: "2f7898458183605f773f03558badc0ea"
server_key_version: "encryptionkey@0"
I20260927 16:39:04.534052 4692 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:39:04.536183 4706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:04.537242 4692 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:39:04.537364 4692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "2dea7a3803074d469ce4260955a64b55"
format_stamp: "Formatted at 2026-09-27 16:39:04 on dist-test-slave-z40b"
server_key: "6ca76076a4da4a42c33ddc953b4e21b6"
server_key_iv: "2f7898458183605f773f03558badc0ea"
server_key_version: "encryptionkey@0"
I20260927 16:39:04.537459 4692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:04.572140 4692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:04.575191 4692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:04.575392 4692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:04.575907 4692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:04.576705 4692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:04.576767 4692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:04.576828 4692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:04.576871 4692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:04.585842 4692 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46205
I20260927 16:39:04.585860 4819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46205 every 8 connection(s)
I20260927 16:39:04.586367 4692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:39:04.587806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 4692
I20260927 16:39:04.587932 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:39:04.588258 392 external_mini_cluster.cc:1428] Setting key 468d4a5c8ef06068e917f6bf11640b9c
Sep 27 16:39:04 dist-test-slave-z40b krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 16:39:04.601127 4536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:04.588149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50525 (local address 127.0.98.62:37873)
0927 16:39:04.588290 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:04.588295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:04.589163 (+ 868us) server_negotiation.cc:407] Connection header received
0927 16:39:04.591059 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:04.591062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:04.591102 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:04.591167 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:04.592351 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.592935 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.594613 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.594801 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.594894 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:04.597457 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:04.597461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:04.597463 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:04.597492 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:04.598878 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:04.599434 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:04.599440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:04.599442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:04.599520 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:04.599851 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:04.599854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:04.599856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:04.600109 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:04.600209 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:04.600750 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 16:39:04.600885 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:39:04.601653 4822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:04.588505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37873 (local address 127.0.98.2:50525)
0927 16:39:04.589025 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:04.589076 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:04.590846 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:04.591348 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:04.591359 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:04.591789 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:04.592217 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:04.592229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.593068 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.593074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:04.594498 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:04.594510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.594965 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.594972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:04.595237 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:04.595777 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:04.595802 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:04.597308 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:04.598991 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:04.598997 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:04.599009 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:04.599337 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:04.599616 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:04.599621 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:04.599623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:04.599754 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:04.600231 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:04.600236 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:04.600456 (+ 220us) client_negotiation.cc:769] Sending connection context
0927 16:39:04.600634 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 16:39:04.600864 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":144,"threads_started":1}
I20260927 16:39:04.602784 4820 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37873
I20260927 16:39:04.603020 4820 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:04.603473 4820 heartbeater.cc:511] Master 127.0.98.62:37873 requested a full tablet report, sending...
I20260927 16:39:04.604429 4497 ts_manager.cc:194] Registered new tserver with Master: 2dea7a3803074d469ce4260955a64b55 (127.0.98.2:46205)
I20260927 16:39:04.605293 4497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:39:04.614159 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37873
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:04.716660 4825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:04.716905 4825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:04.716943 4825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:04.720891 4825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:04.720934 4825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:04.720988 4825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:39:04.725927 4825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:04.726815 4825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:04.727715 4825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:04.736338 4833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:04.736367 4830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:04.736367 4831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:04.736469 4825 server_base.cc:1034] running on GCE node
I20260927 16:39:04.736938 4825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:04.737471 4825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:04.738656 4825 hybrid_clock.cc:648] HybridClock initialized: now 1790527144738647 us; error 27 us; skew 500 ppm
Sep 27 16:39:04 dist-test-slave-z40b krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:04.741400 4825 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:39:04.742102 4825 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:04.742187 4825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:04.742380 4825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:04.744284 4825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-2/data/instance:
uuid: "e8960ca3a79f4a6f867bd00ceb52b3cb"
format_stamp: "Formatted at 2026-09-27 16:39:04 on dist-test-slave-z40b"
server_key: "525924077bc71ca57b70c0683f6dbcbb"
server_key_iv: "ce0aa20aa7d1cf44f84cf1df6180f143"
server_key_version: "encryptionkey@0"
I20260927 16:39:04.744814 4825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e8960ca3a79f4a6f867bd00ceb52b3cb"
format_stamp: "Formatted at 2026-09-27 16:39:04 on dist-test-slave-z40b"
server_key: "525924077bc71ca57b70c0683f6dbcbb"
server_key_iv: "ce0aa20aa7d1cf44f84cf1df6180f143"
server_key_version: "encryptionkey@0"
I20260927 16:39:04.748077 4825 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:39:04.750290 4839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:04.751412 4825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:39:04.751526 4825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "e8960ca3a79f4a6f867bd00ceb52b3cb"
format_stamp: "Formatted at 2026-09-27 16:39:04 on dist-test-slave-z40b"
server_key: "525924077bc71ca57b70c0683f6dbcbb"
server_key_iv: "ce0aa20aa7d1cf44f84cf1df6180f143"
server_key_version: "encryptionkey@0"
I20260927 16:39:04.751634 4825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:04.766371 4825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:04.768828 4825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:04.768997 4825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:04.769515 4825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:04.770223 4825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:04.770290 4825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:04.770354 4825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:04.770399 4825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:04.778597 4825 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35809
I20260927 16:39:04.778623 4952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35809 every 8 connection(s)
I20260927 16:39:04.779099 4825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 16:39:04 dist-test-slave-z40b krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:39:04.788971 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 4825
I20260927 16:39:04.789110 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:39:04.789458 392 external_mini_cluster.cc:1428] Setting key 78730e2d51ed368f515aea4215479691
I20260927 16:39:04.791348 4536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:04.780651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34407 (local address 127.0.98.62:37873)
0927 16:39:04.780811 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:04.780816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:04.781556 (+ 740us) server_negotiation.cc:407] Connection header received
0927 16:39:04.782706 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:04.782709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:04.782754 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:04.782823 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:04.783799 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.784289 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.785278 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.785504 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.785595 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:04.787604 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:04.787607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:04.787609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:04.787631 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:04.789221 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:04.789750 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:04.789752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:04.789770 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:04.789861 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:04.790239 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:04.790242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:04.790244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:04.790482 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:04.790569 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:04.791067 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 16:39:04.791174 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 16:39:04.792032 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:04.780997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37873 (local address 127.0.98.3:34407)
0927 16:39:04.781399 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:04.781435 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:04.782538 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:04.782959 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:04.782965 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:04.783288 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:04.783684 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:04.783695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.784392 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.784394 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:04.785181 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:04.785188 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.785632 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.785635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:04.785756 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:04.786184 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:04.786199 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:04.787470 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:04.789332 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:04.789338 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:04.789352 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:04.789662 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:04.789975 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:04.789977 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:04.789979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:04.790139 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:04.790585 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:04.790593 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:04.790846 (+ 253us) client_negotiation.cc:769] Sending connection context
0927 16:39:04.791008 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 16:39:04.791259 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":103,"threads_started":1}
I20260927 16:39:04.793018 4953 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37873
I20260927 16:39:04.793231 4953 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:04.793620 4953 heartbeater.cc:511] Master 127.0.98.62:37873 requested a full tablet report, sending...
I20260927 16:39:04.794471 4497 ts_manager.cc:194] Registered new tserver with Master: e8960ca3a79f4a6f867bd00ceb52b3cb (127.0.98.3:35809)
I20260927 16:39:04.794953 4497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34407
I20260927 16:39:04.803329 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:39:04.812412 4536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:04.805257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56840 (local address 127.0.98.62:37873)
0927 16:39:04.805347 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:04.805352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:04.805562 (+ 210us) server_negotiation.cc:407] Connection header received
0927 16:39:04.805692 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:04.805695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:04.805734 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:04.805822 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:04.806455 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.806943 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.807807 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.808026 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.808134 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:04.809370 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:04.809373 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:04.809376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:04.809399 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:04.810794 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:04.811234 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:04.811239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:04.811241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:04.811315 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:04.811593 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:04.811597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:04.811599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:04.811786 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:04.811935 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:04.812162 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 16:39:04.812263 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 16:39:04.814813 4497 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:39:04.817608 4497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:39:04.833395 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:04.827044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52494 (local address 127.0.98.2:46205)
0927 16:39:04.829024 (+ 1980us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:04.829038 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:04.829059 (+ 21us) server_negotiation.cc:407] Connection header received
0927 16:39:04.829147 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:04.829152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:04.829318 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:04.829728 (+ 410us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:04.829881 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.830853 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.832079 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.832763 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.833032 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:04.833135 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:04.833226 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 16:39:04.833286 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1333,"thread_start_us":150,"threads_started":1}
I20260927 16:39:04.833448 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:04.826646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46205 (local address 127.0.0.1:52494)
0927 16:39:04.827398 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:04.827412 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:04.827526 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:04.829469 (+ 1943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:04.829472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:04.829488 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:04.829586 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:04.829592 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.830965 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.830968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:04.831981 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:04.831989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.832907 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.832911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:04.833061 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:04.833099 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 16:39:04.833208 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 16:39:04.833312 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":11,"thread_start_us":91,"threads_started":1}
I20260927 16:39:04.833907 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:04.826752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44389 (local address 127.0.0.1:32902)
0927 16:39:04.828137 (+ 1385us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:04.828151 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:04.828236 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:04.828901 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:04.828904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:04.828917 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:04.828987 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:04.828992 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.830426 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.830428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:04.832537 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:04.832549 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.833474 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.833477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:04.833632 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:04.833653 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 16:39:04.833708 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 16:39:04.833778 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1326,"mutex_wait_us":1192,"thread_start_us":83,"threads_started":1}
I20260927 16:39:04.833930 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:04.826075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35809 (local address 127.0.0.1:53528)
0927 16:39:04.827119 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:04.827434 (+ 315us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:04.827556 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:04.828745 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:04.828751 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:04.828798 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:04.828969 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:04.828977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.830425 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.830428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:04.831895 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:04.831904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.832889 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.832892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:04.833657 (+ 765us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:04.833677 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 16:39:04.833740 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 16:39:04.833803 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":73,"thread_start_us":78,"threads_started":1}
I20260927 16:39:04.834291 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:04.827997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53528 (local address 127.0.98.3:35809)
0927 16:39:04.828279 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:04.828291 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:04.828313 (+ 22us) server_negotiation.cc:407] Connection header received
0927 16:39:04.828385 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:04.828391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:04.828583 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:04.828740 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:04.829126 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.830285 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.832017 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.832763 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.832882 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:04.833007 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:04.834079 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0927 16:39:04.834160 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":117,"threads_started":1}
I20260927 16:39:04.834546 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:04.827375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32902 (local address 127.0.98.1:44389)
0927 16:39:04.828005 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:04.828017 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:04.828327 (+ 310us) server_negotiation.cc:407] Connection header received
0927 16:39:04.828405 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:04.828410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:04.828612 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:04.828756 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:04.829116 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.830262 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.832645 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:04.833359 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:04.833452 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:04.833861 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:04.833905 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 16:39:04.834252 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":92,"threads_started":1}
I20260927 16:39:04.836570 4754 tablet_service.cc:1467] Processing CreateTablet for tablet 34077d1369e342cdb766c13c323d3277 (DEFAULT_TABLE table=test-table [id=7c56ae948e2241a28839569fb0b30ce7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:39:04.837225 4887 tablet_service.cc:1467] Processing CreateTablet for tablet 34077d1369e342cdb766c13c323d3277 (DEFAULT_TABLE table=test-table [id=7c56ae948e2241a28839569fb0b30ce7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:39:04.837159 4621 tablet_service.cc:1467] Processing CreateTablet for tablet 34077d1369e342cdb766c13c323d3277 (DEFAULT_TABLE table=test-table [id=7c56ae948e2241a28839569fb0b30ce7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:39:04.837690 4754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34077d1369e342cdb766c13c323d3277. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:04.838183 4621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34077d1369e342cdb766c13c323d3277. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:04.838243 4887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34077d1369e342cdb766c13c323d3277. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:04.844738 4971 tablet_bootstrap.cc:492] T 34077d1369e342cdb766c13c323d3277 P 2dea7a3803074d469ce4260955a64b55: Bootstrap starting.
I20260927 16:39:04.845585 4972 tablet_bootstrap.cc:492] T 34077d1369e342cdb766c13c323d3277 P bef55520bc2c4ceb9ff04aa1f74cdbf1: Bootstrap starting.
I20260927 16:39:04.845657 4970 tablet_bootstrap.cc:492] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb: Bootstrap starting.
I20260927 16:39:04.847106 4971 tablet_bootstrap.cc:654] T 34077d1369e342cdb766c13c323d3277 P 2dea7a3803074d469ce4260955a64b55: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:04.847195 4972 tablet_bootstrap.cc:654] T 34077d1369e342cdb766c13c323d3277 P bef55520bc2c4ceb9ff04aa1f74cdbf1: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:04.847643 4970 tablet_bootstrap.cc:654] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:04.848423 4972 log.cc:824] T 34077d1369e342cdb766c13c323d3277 P bef55520bc2c4ceb9ff04aa1f74cdbf1: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:04.848546 4971 log.cc:824] T 34077d1369e342cdb766c13c323d3277 P 2dea7a3803074d469ce4260955a64b55: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:04.848877 4970 log.cc:824] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:04.850370 4972 tablet_bootstrap.cc:492] T 34077d1369e342cdb766c13c323d3277 P bef55520bc2c4ceb9ff04aa1f74cdbf1: No bootstrap required, opened a new log
I20260927 16:39:04.850461 4971 tablet_bootstrap.cc:492] T 34077d1369e342cdb766c13c323d3277 P 2dea7a3803074d469ce4260955a64b55: No bootstrap required, opened a new log
I20260927 16:39:04.850600 4972 ts_tablet_manager.cc:1397] T 34077d1369e342cdb766c13c323d3277 P bef55520bc2c4ceb9ff04aa1f74cdbf1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:39:04.850641 4971 ts_tablet_manager.cc:1397] T 34077d1369e342cdb766c13c323d3277 P 2dea7a3803074d469ce4260955a64b55: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:39:04.850658 4970 tablet_bootstrap.cc:492] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb: No bootstrap required, opened a new log
I20260927 16:39:04.850782 4970 ts_tablet_manager.cc:1397] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:39:04.853219 4972 raft_consensus.cc:348] T 34077d1369e342cdb766c13c323d3277 P bef55520bc2c4ceb9ff04aa1f74cdbf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8960ca3a79f4a6f867bd00ceb52b3cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35809 } } peers { permanent_uuid: "bef55520bc2c4ceb9ff04aa1f74cdbf1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44389 } } peers { permanent_uuid: "2dea7a3803074d469ce4260955a64b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46205 } }
I20260927 16:39:04.853413 4972 raft_consensus.cc:374] T 34077d1369e342cdb766c13c323d3277 P bef55520bc2c4ceb9ff04aa1f74cdbf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:04.853459 4972 raft_consensus.cc:729] T 34077d1369e342cdb766c13c323d3277 P bef55520bc2c4ceb9ff04aa1f74cdbf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bef55520bc2c4ceb9ff04aa1f74cdbf1, State: Initialized, Role: FOLLOWER
I20260927 16:39:04.853820 4972 consensus_queue.cc:260] T 34077d1369e342cdb766c13c323d3277 P bef55520bc2c4ceb9ff04aa1f74cdbf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8960ca3a79f4a6f867bd00ceb52b3cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35809 } } peers { permanent_uuid: "bef55520bc2c4ceb9ff04aa1f74cdbf1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44389 } } peers { permanent_uuid: "2dea7a3803074d469ce4260955a64b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46205 } }
I20260927 16:39:04.854008 4971 raft_consensus.cc:348] T 34077d1369e342cdb766c13c323d3277 P 2dea7a3803074d469ce4260955a64b55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8960ca3a79f4a6f867bd00ceb52b3cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35809 } } peers { permanent_uuid: "bef55520bc2c4ceb9ff04aa1f74cdbf1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44389 } } peers { permanent_uuid: "2dea7a3803074d469ce4260955a64b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46205 } }
I20260927 16:39:04.854008 4970 raft_consensus.cc:348] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8960ca3a79f4a6f867bd00ceb52b3cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35809 } } peers { permanent_uuid: "bef55520bc2c4ceb9ff04aa1f74cdbf1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44389 } } peers { permanent_uuid: "2dea7a3803074d469ce4260955a64b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46205 } }
I20260927 16:39:04.854283 4970 raft_consensus.cc:374] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:04.854283 4971 raft_consensus.cc:374] T 34077d1369e342cdb766c13c323d3277 P 2dea7a3803074d469ce4260955a64b55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:04.854343 4970 raft_consensus.cc:729] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8960ca3a79f4a6f867bd00ceb52b3cb, State: Initialized, Role: FOLLOWER
I20260927 16:39:04.854343 4971 raft_consensus.cc:729] T 34077d1369e342cdb766c13c323d3277 P 2dea7a3803074d469ce4260955a64b55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dea7a3803074d469ce4260955a64b55, State: Initialized, Role: FOLLOWER
I20260927 16:39:04.854571 4687 heartbeater.cc:503] Master 127.0.98.62:37873 was elected leader, sending a full tablet report...
I20260927 16:39:04.854817 4972 ts_tablet_manager.cc:1428] T 34077d1369e342cdb766c13c323d3277 P bef55520bc2c4ceb9ff04aa1f74cdbf1: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260927 16:39:04.854813 4971 consensus_queue.cc:260] T 34077d1369e342cdb766c13c323d3277 P 2dea7a3803074d469ce4260955a64b55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8960ca3a79f4a6f867bd00ceb52b3cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35809 } } peers { permanent_uuid: "bef55520bc2c4ceb9ff04aa1f74cdbf1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44389 } } peers { permanent_uuid: "2dea7a3803074d469ce4260955a64b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46205 } }
I20260927 16:39:04.855330 4970 consensus_queue.cc:260] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8960ca3a79f4a6f867bd00ceb52b3cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35809 } } peers { permanent_uuid: "bef55520bc2c4ceb9ff04aa1f74cdbf1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44389 } } peers { permanent_uuid: "2dea7a3803074d469ce4260955a64b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46205 } }
I20260927 16:39:04.855443 4820 heartbeater.cc:503] Master 127.0.98.62:37873 was elected leader, sending a full tablet report...
I20260927 16:39:04.855865 4971 ts_tablet_manager.cc:1428] T 34077d1369e342cdb766c13c323d3277 P 2dea7a3803074d469ce4260955a64b55: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 16:39:04.856146 4970 ts_tablet_manager.cc:1428] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 16:39:04.856438 4953 heartbeater.cc:503] Master 127.0.98.62:37873 was elected leader, sending a full tablet report...
W20260927 16:39:04.870362 4688 tablet.cc:2369] T 34077d1369e342cdb766c13c323d3277 P bef55520bc2c4ceb9ff04aa1f74cdbf1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:39:05.029959 4954 tablet.cc:2369] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:39:05.048967 4978 raft_consensus.cc:482] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:39:05.049191 4978 raft_consensus.cc:504] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8960ca3a79f4a6f867bd00ceb52b3cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35809 } } peers { permanent_uuid: "bef55520bc2c4ceb9ff04aa1f74cdbf1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44389 } } peers { permanent_uuid: "2dea7a3803074d469ce4260955a64b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46205 } }
I20260927 16:39:05.050293 4978 leader_election.cc:288] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bef55520bc2c4ceb9ff04aa1f74cdbf1 (127.0.98.1:44389), 2dea7a3803074d469ce4260955a64b55 (127.0.98.2:46205)
I20260927 16:39:05.054229 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:05.050651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55207 (local address 127.0.98.2:46205)
0927 16:39:05.050791 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:05.050796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:05.050897 (+ 101us) server_negotiation.cc:407] Connection header received
0927 16:39:05.051020 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:05.051022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:05.051091 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:05.051173 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:05.051471 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:05.052292 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:05.053271 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:05.053678 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:05.053754 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:05.053826 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:05.054029 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 16:39:05.054107 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 16:39:05.054229 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:05.050581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46205 (local address 127.0.98.3:55207)
0927 16:39:05.050790 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:05.050805 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:05.050908 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:05.051194 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:05.051196 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:05.051254 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:05.051377 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:05.051384 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:05.052384 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:05.052386 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:05.053143 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:05.053150 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:05.053794 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:05.053800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:05.053888 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:05.053914 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 16:39:05.054007 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 16:39:05.054100 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260927 16:39:05.054268 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:05.050650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46823 (local address 127.0.98.1:44389)
0927 16:39:05.050819 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:05.050824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:05.051084 (+ 260us) server_negotiation.cc:407] Connection header received
0927 16:39:05.051156 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:05.051161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:05.051227 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:05.051292 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:05.051581 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:05.052292 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:05.053349 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:05.053713 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:05.053833 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:05.053889 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:05.054101 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 16:39:05.054174 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 16:39:05.054247 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:05.050581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44389 (local address 127.0.98.3:46823)
0927 16:39:05.050996 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:05.051017 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:05.051085 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:05.051318 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:05.051321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:05.051337 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:05.051465 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:05.051471 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:05.052437 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:05.052440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:05.053238 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:05.053247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:05.053850 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:05.053853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:05.053929 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:05.053944 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 16:39:05.054069 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 16:39:05.054128 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":49,"thread_start_us":136,"threads_started":1}
I20260927 16:39:05.054908 4641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34077d1369e342cdb766c13c323d3277" candidate_uuid: "e8960ca3a79f4a6f867bd00ceb52b3cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bef55520bc2c4ceb9ff04aa1f74cdbf1" is_pre_election: true
I20260927 16:39:05.055001 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34077d1369e342cdb766c13c323d3277" candidate_uuid: "e8960ca3a79f4a6f867bd00ceb52b3cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dea7a3803074d469ce4260955a64b55" is_pre_election: true
I20260927 16:39:05.055193 4641 raft_consensus.cc:2454] T 34077d1369e342cdb766c13c323d3277 P bef55520bc2c4ceb9ff04aa1f74cdbf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8960ca3a79f4a6f867bd00ceb52b3cb in term 0.
I20260927 16:39:05.055346 4774 raft_consensus.cc:2454] T 34077d1369e342cdb766c13c323d3277 P 2dea7a3803074d469ce4260955a64b55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8960ca3a79f4a6f867bd00ceb52b3cb in term 0.
I20260927 16:39:05.055711 4840 leader_election.cc:302] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bef55520bc2c4ceb9ff04aa1f74cdbf1, e8960ca3a79f4a6f867bd00ceb52b3cb; no voters:
I20260927 16:39:05.055998 4978 raft_consensus.cc:2781] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:39:05.056141 4978 raft_consensus.cc:482] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:39:05.056195 4978 raft_consensus.cc:3037] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:05.057225 4978 raft_consensus.cc:504] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8960ca3a79f4a6f867bd00ceb52b3cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35809 } } peers { permanent_uuid: "bef55520bc2c4ceb9ff04aa1f74cdbf1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44389 } } peers { permanent_uuid: "2dea7a3803074d469ce4260955a64b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46205 } }
I20260927 16:39:05.057583 4978 leader_election.cc:288] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb [CANDIDATE]: Term 1 election: Requested vote from peers bef55520bc2c4ceb9ff04aa1f74cdbf1 (127.0.98.1:44389), 2dea7a3803074d469ce4260955a64b55 (127.0.98.2:46205)
I20260927 16:39:05.057880 4641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34077d1369e342cdb766c13c323d3277" candidate_uuid: "e8960ca3a79f4a6f867bd00ceb52b3cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bef55520bc2c4ceb9ff04aa1f74cdbf1"
I20260927 16:39:05.058010 4641 raft_consensus.cc:3037] T 34077d1369e342cdb766c13c323d3277 P bef55520bc2c4ceb9ff04aa1f74cdbf1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:05.059247 4641 raft_consensus.cc:2454] T 34077d1369e342cdb766c13c323d3277 P bef55520bc2c4ceb9ff04aa1f74cdbf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8960ca3a79f4a6f867bd00ceb52b3cb in term 1.
I20260927 16:39:05.059401 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34077d1369e342cdb766c13c323d3277" candidate_uuid: "e8960ca3a79f4a6f867bd00ceb52b3cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dea7a3803074d469ce4260955a64b55"
I20260927 16:39:05.059516 4774 raft_consensus.cc:3037] T 34077d1369e342cdb766c13c323d3277 P 2dea7a3803074d469ce4260955a64b55 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:05.059573 4840 leader_election.cc:302] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bef55520bc2c4ceb9ff04aa1f74cdbf1, e8960ca3a79f4a6f867bd00ceb52b3cb; no voters:
I20260927 16:39:05.059753 4978 raft_consensus.cc:2781] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:39:05.059990 4978 raft_consensus.cc:686] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb [term 1 LEADER]: Becoming Leader. State: Replica: e8960ca3a79f4a6f867bd00ceb52b3cb, State: Running, Role: LEADER
I20260927 16:39:05.060341 4978 consensus_queue.cc:237] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8960ca3a79f4a6f867bd00ceb52b3cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35809 } } peers { permanent_uuid: "bef55520bc2c4ceb9ff04aa1f74cdbf1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44389 } } peers { permanent_uuid: "2dea7a3803074d469ce4260955a64b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46205 } }
I20260927 16:39:05.063532 4494 catalog_manager.cc:5495] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb reported cstate change: term changed from 0 to 1, leader changed from <none> to e8960ca3a79f4a6f867bd00ceb52b3cb (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e8960ca3a79f4a6f867bd00ceb52b3cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8960ca3a79f4a6f867bd00ceb52b3cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bef55520bc2c4ceb9ff04aa1f74cdbf1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dea7a3803074d469ce4260955a64b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46205 } health_report { overall_health: UNKNOWN } } }
I20260927 16:39:05.080871 4774 raft_consensus.cc:2454] T 34077d1369e342cdb766c13c323d3277 P 2dea7a3803074d469ce4260955a64b55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8960ca3a79f4a6f867bd00ceb52b3cb in term 1.
W20260927 16:39:05.087379 4821 tablet.cc:2369] T 34077d1369e342cdb766c13c323d3277 P 2dea7a3803074d469ce4260955a64b55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:39:05.124759 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:05.121012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53542 (local address 127.0.98.3:35809)
0927 16:39:05.121143 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:05.121146 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:05.121160 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:39:05.121301 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:05.121303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:05.121358 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:05.121442 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:39:05.121688 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:05.122187 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:05.123397 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:05.123608 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:05.123688 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:05.123974 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:39:05.124341 (+ 367us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:39:05.124438 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:05.124533 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 16:39:05.124588 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:39:05.130200 4774 raft_consensus.cc:1261] T 34077d1369e342cdb766c13c323d3277 P 2dea7a3803074d469ce4260955a64b55 [term 1 FOLLOWER]: Refusing update from remote peer e8960ca3a79f4a6f867bd00ceb52b3cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:39:05.130210 4641 raft_consensus.cc:1261] T 34077d1369e342cdb766c13c323d3277 P bef55520bc2c4ceb9ff04aa1f74cdbf1 [term 1 FOLLOWER]: Refusing update from remote peer e8960ca3a79f4a6f867bd00ceb52b3cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:39:05.130851 4980 consensus_queue.cc:1035] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bef55520bc2c4ceb9ff04aa1f74cdbf1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:39:05.131031 4978 consensus_queue.cc:1035] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dea7a3803074d469ce4260955a64b55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:39:05.139345 4984 mvcc.cc:205] Tried to move back new op lower bound from 7333999186446606336 to 7333999186169716736. Current Snapshot: MvccSnapshot[applied={T|T < 7333999186446606336}]
I20260927 16:39:05.139667 4985 mvcc.cc:205] Tried to move back new op lower bound from 7333999186446606336 to 7333999186169716736. Current Snapshot: MvccSnapshot[applied={T|T < 7333999186446606336}]
I20260927 16:39:05.147466 4495 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:39:05.147614 4495 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:39:05.149758 4495 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 10abab21879e48938901c8eabbeae6d2: Sending DeleteTablet for 3 replicas of tablet 34077d1369e342cdb766c13c323d3277
I20260927 16:39:05.150465 4887 tablet_service.cc:1514] Processing DeleteTablet for tablet 34077d1369e342cdb766c13c323d3277 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:39:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53528
I20260927 16:39:05.150491 4754 tablet_service.cc:1514] Processing DeleteTablet for tablet 34077d1369e342cdb766c13c323d3277 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:39:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52494
I20260927 16:39:05.151021 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 4559
I20260927 16:39:05.151036 4621 tablet_service.cc:1514] Processing DeleteTablet for tablet 34077d1369e342cdb766c13c323d3277 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:39:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32902
I20260927 16:39:05.151610 4993 tablet_replica.cc:330] T 34077d1369e342cdb766c13c323d3277 P 2dea7a3803074d469ce4260955a64b55: stopping tablet replica
I20260927 16:39:05.151875 4993 raft_consensus.cc:2229] T 34077d1369e342cdb766c13c323d3277 P 2dea7a3803074d469ce4260955a64b55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:39:05.152168 4993 raft_consensus.cc:2258] T 34077d1369e342cdb766c13c323d3277 P 2dea7a3803074d469ce4260955a64b55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:39:05.152226 4992 tablet_replica.cc:330] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb: stopping tablet replica
I20260927 16:39:05.152494 4992 raft_consensus.cc:2229] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:39:05.152839 4992 raft_consensus.cc:2258] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:39:05.154215 4993 ts_tablet_manager.cc:1905] T 34077d1369e342cdb766c13c323d3277 P 2dea7a3803074d469ce4260955a64b55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:39:05.156034 4992 ts_tablet_manager.cc:1905] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:39:05.156838 4993 ts_tablet_manager.cc:1918] T 34077d1369e342cdb766c13c323d3277 P 2dea7a3803074d469ce4260955a64b55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:39:05.156925 4993 log.cc:1196] T 34077d1369e342cdb766c13c323d3277 P 2dea7a3803074d469ce4260955a64b55: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-1/wal/wals/34077d1369e342cdb766c13c323d3277
I20260927 16:39:05.157267 4993 ts_tablet_manager.cc:1939] T 34077d1369e342cdb766c13c323d3277 P 2dea7a3803074d469ce4260955a64b55: Deleting consensus metadata
W20260927 16:39:05.159521 4481 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44389 (error 108)
I20260927 16:39:05.160468 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:05.160182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44389 (local address 127.0.0.1:32910)
0927 16:39:05.160300 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:05.160372 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 16:39:05.160538 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 4692
W20260927 16:39:05.160737 4481 catalog_manager.cc:4584] TS bef55520bc2c4ceb9ff04aa1f74cdbf1 (127.0.98.1:44389): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 34077d1369e342cdb766c13c323d3277: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44389: connect: Connection refused (error 111)
I20260927 16:39:05.190759 4992 ts_tablet_manager.cc:1918] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:39:05.190929 4992 log.cc:1196] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790527121515808-392-0/minicluster-data/ts-2/wal/wals/34077d1369e342cdb766c13c323d3277
I20260927 16:39:05.191320 4992 ts_tablet_manager.cc:1939] T 34077d1369e342cdb766c13c323d3277 P e8960ca3a79f4a6f867bd00ceb52b3cb: Deleting consensus metadata
I20260927 16:39:05.196815 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 4825
I20260927 16:39:05.198796 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:05.198544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46205 (local address 127.0.0.1:52508)
0927 16:39:05.198667 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:05.198721 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 16:39:05.213714 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:05.213502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44389 (local address 127.0.0.1:32920)
0927 16:39:05.213608 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:05.213660 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 16:39:05.214298 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:05.214133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44389 (local address 127.0.0.1:32934)
0927 16:39:05.214215 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:05.214254 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 16:39:05.220747 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 4466
2026-09-27T16:39:05Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2390 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T16:39:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:39:05Z Disabled control of system clock
I20260927 16:39:05.251189 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46733
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 16:39:05.360028 4998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:05.360257 4998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:05.360291 4998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:05.364243 4998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:39:05.364289 4998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:05.364305 4998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:05.364321 4998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:39:05.364387 4998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:39:05.368978 4998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46733
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:05.369932 4998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:05.371243 4998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:05.379176 5006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:05.379197 5004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:05.379259 4998 server_base.cc:1034] running on GCE node
W20260927 16:39:05.379262 5003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:05.379961 4998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:05.380972 4998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:05.382151 4998 hybrid_clock.cc:648] HybridClock initialized: now 1790527145382123 us; error 41 us; skew 500 ppm
I20260927 16:39:05.383803 4998 webserver.cc:460] Webserver started at http://127.0.98.62:38687/ using document root <none> and password file <none>
I20260927 16:39:05.384320 4998 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:05.384406 4998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:05.384665 4998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:05.386605 4998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "d4ab2c62a94f40aa90ab5aa9a0164fc4"
format_stamp: "Formatted at 2026-09-27 16:39:05 on dist-test-slave-z40b"
server_key: "1e4691bd91ea3805daf114de4cd8cb60"
server_key_iv: "e1e64f84c1ca7546c25c5c9cda8984cb"
server_key_version: "encryptionkey@0"
I20260927 16:39:05.387166 4998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "d4ab2c62a94f40aa90ab5aa9a0164fc4"
format_stamp: "Formatted at 2026-09-27 16:39:05 on dist-test-slave-z40b"
server_key: "1e4691bd91ea3805daf114de4cd8cb60"
server_key_iv: "e1e64f84c1ca7546c25c5c9cda8984cb"
server_key_version: "encryptionkey@0"
I20260927 16:39:05.390550 4998 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260927 16:39:05.392815 5012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:05.393899 4998 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:39:05.394023 4998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "d4ab2c62a94f40aa90ab5aa9a0164fc4"
format_stamp: "Formatted at 2026-09-27 16:39:05 on dist-test-slave-z40b"
server_key: "1e4691bd91ea3805daf114de4cd8cb60"
server_key_iv: "e1e64f84c1ca7546c25c5c9cda8984cb"
server_key_version: "encryptionkey@0"
I20260927 16:39:05.394130 4998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:05.398379 4998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:05.398942 4998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:05.399103 4998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:05.405481 5064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46733 every 8 connection(s)
I20260927 16:39:05.405534 4998 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46733
I20260927 16:39:05.406306 4998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:39:05.409226 5065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:05.413978 5065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4ab2c62a94f40aa90ab5aa9a0164fc4: Bootstrap starting.
I20260927 16:39:05.416152 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 4998
I20260927 16:39:05.416283 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:39:05.416240 5065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4ab2c62a94f40aa90ab5aa9a0164fc4: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:05.416546 392 external_mini_cluster.cc:1428] Setting key 346cbb97bbc0122ff0db3ef466f2e14a
I20260927 16:39:05.416982 5065 log.cc:824] T 00000000000000000000000000000000 P d4ab2c62a94f40aa90ab5aa9a0164fc4: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:05.418784 5065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4ab2c62a94f40aa90ab5aa9a0164fc4: No bootstrap required, opened a new log
I20260927 16:39:05.421273 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:05.417586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45082 (local address 127.0.98.62:46733)
0927 16:39:05.418137 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:05.418150 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:05.418181 (+ 31us) server_negotiation.cc:407] Connection header received
0927 16:39:05.418730 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:05.418744 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:05.419036 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:05.419400 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:05.420146 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:05.420150 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:05.420160 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:39:05.420170 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:05.420221 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:39:05.420235 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:05.420345 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:05.420446 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 16:39:05.420675 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":118,"threads_started":1}
I20260927 16:39:05.422657 5065 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4ab2c62a94f40aa90ab5aa9a0164fc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4ab2c62a94f40aa90ab5aa9a0164fc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46733 } }
I20260927 16:39:05.422888 5065 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4ab2c62a94f40aa90ab5aa9a0164fc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:05.422968 5065 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4ab2c62a94f40aa90ab5aa9a0164fc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4ab2c62a94f40aa90ab5aa9a0164fc4, State: Initialized, Role: FOLLOWER
I20260927 16:39:05.423429 5065 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4ab2c62a94f40aa90ab5aa9a0164fc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4ab2c62a94f40aa90ab5aa9a0164fc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46733 } }
I20260927 16:39:05.423571 5065 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4ab2c62a94f40aa90ab5aa9a0164fc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:39:05.423635 5065 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4ab2c62a94f40aa90ab5aa9a0164fc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:39:05.423744 5065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4ab2c62a94f40aa90ab5aa9a0164fc4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:05.450263 5065 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4ab2c62a94f40aa90ab5aa9a0164fc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4ab2c62a94f40aa90ab5aa9a0164fc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46733 } }
I20260927 16:39:05.450724 5065 leader_election.cc:302] T 00000000000000000000000000000000 P d4ab2c62a94f40aa90ab5aa9a0164fc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4ab2c62a94f40aa90ab5aa9a0164fc4; no voters:
I20260927 16:39:05.451133 5065 leader_election.cc:288] T 00000000000000000000000000000000 P d4ab2c62a94f40aa90ab5aa9a0164fc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:39:05.451257 5070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4ab2c62a94f40aa90ab5aa9a0164fc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:39:05.451495 5070 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4ab2c62a94f40aa90ab5aa9a0164fc4 [term 1 LEADER]: Becoming Leader. State: Replica: d4ab2c62a94f40aa90ab5aa9a0164fc4, State: Running, Role: LEADER
I20260927 16:39:05.451812 5070 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4ab2c62a94f40aa90ab5aa9a0164fc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4ab2c62a94f40aa90ab5aa9a0164fc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46733 } }
I20260927 16:39:05.452524 5065 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4ab2c62a94f40aa90ab5aa9a0164fc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:39:05.453306 5071 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4ab2c62a94f40aa90ab5aa9a0164fc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4ab2c62a94f40aa90ab5aa9a0164fc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4ab2c62a94f40aa90ab5aa9a0164fc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46733 } } }
I20260927 16:39:05.453429 5071 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4ab2c62a94f40aa90ab5aa9a0164fc4 [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:05.453823 5072 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4ab2c62a94f40aa90ab5aa9a0164fc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4ab2c62a94f40aa90ab5aa9a0164fc4. Latest consensus state: current_term: 1 leader_uuid: "d4ab2c62a94f40aa90ab5aa9a0164fc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4ab2c62a94f40aa90ab5aa9a0164fc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46733 } } }
I20260927 16:39:05.453956 5072 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4ab2c62a94f40aa90ab5aa9a0164fc4 [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:05.454346 5077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:39:05.457482 5077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:39:05.462807 5077 catalog_manager.cc:1347] Generated new cluster ID: 3d1c2ca3ad554a1bbb76468d3133b98c
I20260927 16:39:05.462862 5077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:39:05.478822 5077 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:39:05.479118 5077 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:39:05.484225 5077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4ab2c62a94f40aa90ab5aa9a0164fc4: Generated new TSK 0
I20260927 16:39:05.484767 5077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:39:05.488775 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46733
--builtin_ntp_servers=127.0.98.20:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 16:39:05.596609 5089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:05.596824 5089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:05.596868 5089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:05.600854 5089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:05.600903 5089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:05.600955 5089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:39:05.605472 5089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:05.606351 5089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:05.607594 5089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:05.616525 5094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:05.616534 5097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:05.616566 5089 server_base.cc:1034] running on GCE node
W20260927 16:39:05.616580 5095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:05.617082 5089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:05.617654 5089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:05.618834 5089 hybrid_clock.cc:648] HybridClock initialized: now 1790527145618822 us; error 22 us; skew 500 ppm
I20260927 16:39:05.620566 5089 webserver.cc:460] Webserver started at http://127.0.98.1:46403/ using document root <none> and password file <none>
I20260927 16:39:05.621145 5089 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:05.621232 5089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:05.621456 5089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:05.623431 5089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-0/data/instance:
uuid: "3ba42588b4354d13991e7f21ba75df3f"
format_stamp: "Formatted at 2026-09-27 16:39:05 on dist-test-slave-z40b"
server_key: "6de2ed6246d7649276d9de3475133876"
server_key_iv: "53b7d9e0ff166503b58cfdab7eec4f65"
server_key_version: "encryptionkey@0"
I20260927 16:39:05.624042 5089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3ba42588b4354d13991e7f21ba75df3f"
format_stamp: "Formatted at 2026-09-27 16:39:05 on dist-test-slave-z40b"
server_key: "6de2ed6246d7649276d9de3475133876"
server_key_iv: "53b7d9e0ff166503b58cfdab7eec4f65"
server_key_version: "encryptionkey@0"
I20260927 16:39:05.627446 5089 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:39:05.629706 5103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:05.630803 5089 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:05.630929 5089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "3ba42588b4354d13991e7f21ba75df3f"
format_stamp: "Formatted at 2026-09-27 16:39:05 on dist-test-slave-z40b"
server_key: "6de2ed6246d7649276d9de3475133876"
server_key_iv: "53b7d9e0ff166503b58cfdab7eec4f65"
server_key_version: "encryptionkey@0"
I20260927 16:39:05.631031 5089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:05.635412 5089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:05.635949 5089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:05.636130 5089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:05.636642 5089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:05.637437 5089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:05.637504 5089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:05.637565 5089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:05.637614 5089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:05.645922 5089 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34575
I20260927 16:39:05.645942 5216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34575 every 8 connection(s)
I20260927 16:39:05.646697 5089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:39:05.652252 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:05.648369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43343 (local address 127.0.98.62:46733)
0927 16:39:05.648510 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:05.648515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:05.649482 (+ 967us) server_negotiation.cc:407] Connection header received
0927 16:39:05.650742 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:05.650747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:05.650799 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:05.650874 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:05.651894 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:05.651896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:05.651898 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:39:05.651909 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:05.651930 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:39:05.651937 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:05.652011 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:05.652119 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 16:39:05.652154 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:39:05.653070 5219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:05.648747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46733 (local address 127.0.98.1:43343)
0927 16:39:05.649345 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:05.649384 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:05.650512 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:05.651015 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:05.651021 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:05.651101 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:05.651670 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:39:05.651671 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:05.651706 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:39:05.651765 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:05.652010 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:05.652016 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:05.652019 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 16:39:05.652137 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 16:39:05.652298 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":111,"threads_started":1}
I20260927 16:39:05.654201 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 5089
I20260927 16:39:05.654315 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:39:05.654341 5217 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46733
I20260927 16:39:05.654513 5217 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:05.654639 392 external_mini_cluster.cc:1428] Setting key 47c8c7486cfd4eb85cf3f41e5f39125c
I20260927 16:39:05.654816 5217 heartbeater.cc:511] Master 127.0.98.62:46733 requested a full tablet report, sending...
I20260927 16:39:05.656322 5028 ts_manager.cc:194] Registered new tserver with Master: 3ba42588b4354d13991e7f21ba75df3f (127.0.98.1:34575)
I20260927 16:39:05.656736 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46733
--builtin_ntp_servers=127.0.98.20:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 16:39:05.764648 5220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:05.764881 5220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:05.764916 5220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:05.768623 5220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:05.768659 5220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:05.768707 5220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:39:05.773177 5220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:05.774016 5220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:05.775241 5220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:05.784073 5226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:05.784057 5225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:05.784312 5220 server_base.cc:1034] running on GCE node
W20260927 16:39:05.784090 5228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:05.784832 5220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:05.785420 5220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:05.786633 5220 hybrid_clock.cc:648] HybridClock initialized: now 1790527145786611 us; error 42 us; skew 500 ppm
I20260927 16:39:05.788368 5220 webserver.cc:460] Webserver started at http://127.0.98.2:43861/ using document root <none> and password file <none>
I20260927 16:39:05.788918 5220 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:05.789005 5220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:05.789224 5220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:05.791584 5220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-1/data/instance:
uuid: "2aad15c492324159877587873a4c75d5"
format_stamp: "Formatted at 2026-09-27 16:39:05 on dist-test-slave-z40b"
server_key: "ff4679ce1f57edd2acbf29825455b89e"
server_key_iv: "39a6c815c6d6cfa510c0849c9dcd73be"
server_key_version: "encryptionkey@0"
I20260927 16:39:05.792153 5220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2aad15c492324159877587873a4c75d5"
format_stamp: "Formatted at 2026-09-27 16:39:05 on dist-test-slave-z40b"
server_key: "ff4679ce1f57edd2acbf29825455b89e"
server_key_iv: "39a6c815c6d6cfa510c0849c9dcd73be"
server_key_version: "encryptionkey@0"
I20260927 16:39:05.795574 5220 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:39:05.797726 5234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:05.798854 5220 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:05.798978 5220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "2aad15c492324159877587873a4c75d5"
format_stamp: "Formatted at 2026-09-27 16:39:05 on dist-test-slave-z40b"
server_key: "ff4679ce1f57edd2acbf29825455b89e"
server_key_iv: "39a6c815c6d6cfa510c0849c9dcd73be"
server_key_version: "encryptionkey@0"
I20260927 16:39:05.799086 5220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:05.803351 5220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:05.803882 5220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:05.804064 5220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:05.804549 5220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:05.805456 5220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:05.805533 5220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:05.805606 5220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:05.805660 5220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:05.814612 5220 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34527
I20260927 16:39:05.814658 5347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34527 every 8 connection(s)
I20260927 16:39:05.815483 5220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:39:05.820940 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:05.817128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34409 (local address 127.0.98.62:46733)
0927 16:39:05.817273 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:05.817277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:05.818230 (+ 953us) server_negotiation.cc:407] Connection header received
0927 16:39:05.819458 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:05.819460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:05.819496 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:05.819571 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:05.820593 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:05.820595 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:05.820596 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:39:05.820605 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:05.820619 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:39:05.820625 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:05.820690 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:05.820789 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 16:39:05.820822 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:39:05.821712 5350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:05.817483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46733 (local address 127.0.98.2:34409)
0927 16:39:05.818106 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:05.818140 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:05.819252 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:05.819711 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:05.819717 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:05.819798 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:05.820371 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:39:05.820372 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:05.820408 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:39:05.820468 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:05.820678 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:05.820682 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:05.820685 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 16:39:05.820831 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 16:39:05.821026 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":121,"threads_started":1}
I20260927 16:39:05.822388 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 5220
I20260927 16:39:05.822525 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:39:05.822808 5348 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46733
I20260927 16:39:05.822827 392 external_mini_cluster.cc:1428] Setting key d56c53e4357dc7f8869503a87e7f92b4
I20260927 16:39:05.823005 5348 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:05.823432 5348 heartbeater.cc:511] Master 127.0.98.62:46733 requested a full tablet report, sending...
I20260927 16:39:05.824441 5028 ts_manager.cc:194] Registered new tserver with Master: 2aad15c492324159877587873a4c75d5 (127.0.98.2:34527)
I20260927 16:39:05.825080 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46733
--builtin_ntp_servers=127.0.98.20:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 16:39:05.935320 5351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:05.935549 5351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:05.935603 5351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:05.939505 5351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:05.939554 5351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:05.939607 5351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:39:05.944195 5351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:05.945120 5351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:05.946422 5351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:05.955818 5359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:05.955835 5357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:05.955807 5356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:05.956075 5351 server_base.cc:1034] running on GCE node
I20260927 16:39:05.956485 5351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:05.957003 5351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:05.958154 5351 hybrid_clock.cc:648] HybridClock initialized: now 1790527145958141 us; error 23 us; skew 500 ppm
I20260927 16:39:05.959882 5351 webserver.cc:460] Webserver started at http://127.0.98.3:42625/ using document root <none> and password file <none>
I20260927 16:39:05.960450 5351 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:05.960507 5351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:05.960748 5351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:05.962627 5351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-2/data/instance:
uuid: "745f1209d02c45adb1f00ab810e9a40e"
format_stamp: "Formatted at 2026-09-27 16:39:05 on dist-test-slave-z40b"
server_key: "9c79c765e5eabcde820ac25a88e8e229"
server_key_iv: "ded3881ed6c9e011904c1faf78215b05"
server_key_version: "encryptionkey@0"
I20260927 16:39:05.963194 5351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-2/wal/instance:
uuid: "745f1209d02c45adb1f00ab810e9a40e"
format_stamp: "Formatted at 2026-09-27 16:39:05 on dist-test-slave-z40b"
server_key: "9c79c765e5eabcde820ac25a88e8e229"
server_key_iv: "ded3881ed6c9e011904c1faf78215b05"
server_key_version: "encryptionkey@0"
I20260927 16:39:05.966676 5351 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 16:39:05.968825 5365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:05.969905 5351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 16:39:05.970009 5351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "745f1209d02c45adb1f00ab810e9a40e"
format_stamp: "Formatted at 2026-09-27 16:39:05 on dist-test-slave-z40b"
server_key: "9c79c765e5eabcde820ac25a88e8e229"
server_key_iv: "ded3881ed6c9e011904c1faf78215b05"
server_key_version: "encryptionkey@0"
I20260927 16:39:05.970108 5351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:05.974568 5351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:05.975067 5351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:05.975286 5351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:05.975792 5351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:05.976639 5351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:05.976678 5351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:05.976737 5351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:05.976775 5351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:05.985780 5351 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43539
I20260927 16:39:05.985816 5478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43539 every 8 connection(s)
I20260927 16:39:05.986620 5351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:39:05.990773 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 5351
I20260927 16:39:05.990908 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:39:05.991168 392 external_mini_cluster.cc:1428] Setting key b653ed4fcfc096f4a820e870a2c2c803
I20260927 16:39:05.992285 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:05.988202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36955 (local address 127.0.98.62:46733)
0927 16:39:05.988318 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:05.988321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:05.989286 (+ 965us) server_negotiation.cc:407] Connection header received
0927 16:39:05.990531 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:05.990534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:05.990581 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:05.990660 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:05.991845 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:05.991848 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:05.991851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:39:05.991862 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:05.991883 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:39:05.991892 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:05.991981 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:05.992148 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 16:39:05.992191 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 16:39:05.993202 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:05.988545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46733 (local address 127.0.98.3:36955)
0927 16:39:05.989158 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:05.989193 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:05.990332 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:05.990799 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:05.990805 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:05.990917 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:05.991547 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:39:05.991549 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:05.991585 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:39:05.991671 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:05.991979 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:05.991990 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:05.991999 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 16:39:05.992190 (+ 191us) client_negotiation.cc:240] Negotiation successful
0927 16:39:05.992415 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":101,"threads_started":1}
I20260927 16:39:05.994133 5479 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46733
I20260927 16:39:05.994339 5479 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:05.994692 5479 heartbeater.cc:511] Master 127.0.98.62:46733 requested a full tablet report, sending...
I20260927 16:39:05.995595 5028 ts_manager.cc:194] Registered new tserver with Master: 745f1209d02c45adb1f00ab810e9a40e (127.0.98.3:43539)
I20260927 16:39:06.004664 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:39:06.007720 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:06.006394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49738 (local address 127.0.98.62:46733)
0927 16:39:06.006503 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:06.006506 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:06.006666 (+ 160us) server_negotiation.cc:407] Connection header received
0927 16:39:06.006756 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:06.006757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:06.006782 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:06.006843 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:06.007460 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:06.007462 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:06.007464 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:39:06.007474 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:06.007490 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:39:06.007497 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:06.007562 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:06.007606 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 16:39:06.007634 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:39:06.010172 5028 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:39:06.013131 5028 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:39:06.024319 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:06.020959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43539 (local address 127.0.0.1:53160)
0927 16:39:06.021718 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:06.021762 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:06.021853 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:06.022753 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:06.022756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:06.022770 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:06.023244 (+ 474us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:39:06.023246 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:06.023264 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:39:06.023323 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:06.024087 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:06.024092 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:06.024097 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 16:39:06.024169 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 16:39:06.024196 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":87,"threads_started":1}
I20260927 16:39:06.024413 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:06.021076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53160 (local address 127.0.98.3:43539)
0927 16:39:06.022368 (+ 1292us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:06.022377 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:06.022395 (+ 18us) server_negotiation.cc:407] Connection header received
0927 16:39:06.022438 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:06.022443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:06.022613 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:06.022963 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:06.023870 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:06.023874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:06.023882 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:39:06.023889 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:06.023938 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:39:06.023960 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:06.024098 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:06.024195 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 16:39:06.024259 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1208,"thread_start_us":92,"threads_started":1}
I20260927 16:39:06.024837 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:06.021326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34527 (local address 127.0.0.1:41030)
0927 16:39:06.022058 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:06.022082 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:06.022158 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:06.023298 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:06.023301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:06.023314 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:06.023656 (+ 342us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:39:06.023658 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:06.023667 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:39:06.023691 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:06.024045 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:06.024050 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:06.024053 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 16:39:06.024130 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 16:39:06.024731 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":543,"thread_start_us":88,"threads_started":1}
I20260927 16:39:06.024312 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:06.021086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34575 (local address 127.0.0.1:58678)
0927 16:39:06.022028 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:06.022055 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:06.022119 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:06.022598 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:06.022604 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:06.022673 (+ 69us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:06.023354 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:39:06.023355 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:06.023363 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:39:06.023385 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:06.023962 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:06.023970 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:06.023978 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 16:39:06.024089 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 16:39:06.024189 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"thread_start_us":65,"threads_started":1}
I20260927 16:39:06.025120 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:06.021513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41030 (local address 127.0.98.2:34527)
0927 16:39:06.021896 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:06.021922 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:06.022179 (+ 257us) server_negotiation.cc:407] Connection header received
0927 16:39:06.022250 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:06.022256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:06.022439 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:06.022603 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:06.023816 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:06.023822 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:06.023831 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:39:06.023842 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:06.023889 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:39:06.023919 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:06.024006 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:06.024220 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 16:39:06.024967 (+ 747us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":102,"threads_started":1}
I20260927 16:39:06.025975 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:06.021714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58678 (local address 127.0.98.1:34575)
0927 16:39:06.022082 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:06.022095 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:06.022121 (+ 26us) server_negotiation.cc:407] Connection header received
0927 16:39:06.022254 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:06.022260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:06.022422 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:06.022597 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:06.023714 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:06.023722 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:06.023736 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:39:06.023768 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:06.023810 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:39:06.023845 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:06.023981 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:06.024561 (+ 580us) server_negotiation.cc:299] Negotiation successful
0927 16:39:06.024764 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":108,"threads_started":1}
I20260927 16:39:06.027402 5400 tablet_service.cc:1467] Processing CreateTablet for tablet c0b2189e6edc411695dd7a5f3742c90d (DEFAULT_TABLE table=test-table [id=b6b01eb1ffa14233a2bbb32a5ea152ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:39:06.027724 5282 tablet_service.cc:1467] Processing CreateTablet for tablet c0b2189e6edc411695dd7a5f3742c90d (DEFAULT_TABLE table=test-table [id=b6b01eb1ffa14233a2bbb32a5ea152ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:39:06.028424 5400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b2189e6edc411695dd7a5f3742c90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:06.028653 5282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b2189e6edc411695dd7a5f3742c90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:06.028402 5151 tablet_service.cc:1467] Processing CreateTablet for tablet c0b2189e6edc411695dd7a5f3742c90d (DEFAULT_TABLE table=test-table [id=b6b01eb1ffa14233a2bbb32a5ea152ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:39:06.029241 5151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b2189e6edc411695dd7a5f3742c90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:06.035797 5497 tablet_bootstrap.cc:492] T c0b2189e6edc411695dd7a5f3742c90d P 2aad15c492324159877587873a4c75d5: Bootstrap starting.
I20260927 16:39:06.036273 5498 tablet_bootstrap.cc:492] T c0b2189e6edc411695dd7a5f3742c90d P 3ba42588b4354d13991e7f21ba75df3f: Bootstrap starting.
I20260927 16:39:06.037905 5497 tablet_bootstrap.cc:654] T c0b2189e6edc411695dd7a5f3742c90d P 2aad15c492324159877587873a4c75d5: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:06.038084 5498 tablet_bootstrap.cc:654] T c0b2189e6edc411695dd7a5f3742c90d P 3ba42588b4354d13991e7f21ba75df3f: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:06.038713 5498 log.cc:824] T c0b2189e6edc411695dd7a5f3742c90d P 3ba42588b4354d13991e7f21ba75df3f: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:06.038714 5497 log.cc:824] T c0b2189e6edc411695dd7a5f3742c90d P 2aad15c492324159877587873a4c75d5: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:06.039958 5496 tablet_bootstrap.cc:492] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e: Bootstrap starting.
I20260927 16:39:06.040812 5497 tablet_bootstrap.cc:492] T c0b2189e6edc411695dd7a5f3742c90d P 2aad15c492324159877587873a4c75d5: No bootstrap required, opened a new log
I20260927 16:39:06.040818 5498 tablet_bootstrap.cc:492] T c0b2189e6edc411695dd7a5f3742c90d P 3ba42588b4354d13991e7f21ba75df3f: No bootstrap required, opened a new log
I20260927 16:39:06.040980 5498 ts_tablet_manager.cc:1397] T c0b2189e6edc411695dd7a5f3742c90d P 3ba42588b4354d13991e7f21ba75df3f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 16:39:06.041090 5497 ts_tablet_manager.cc:1397] T c0b2189e6edc411695dd7a5f3742c90d P 2aad15c492324159877587873a4c75d5: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 16:39:06.041940 5496 tablet_bootstrap.cc:654] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:06.042716 5496 log.cc:824] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:06.044678 5496 tablet_bootstrap.cc:492] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e: No bootstrap required, opened a new log
I20260927 16:39:06.044669 5497 raft_consensus.cc:348] T c0b2189e6edc411695dd7a5f3742c90d P 2aad15c492324159877587873a4c75d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745f1209d02c45adb1f00ab810e9a40e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43539 } } peers { permanent_uuid: "3ba42588b4354d13991e7f21ba75df3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34575 } } peers { permanent_uuid: "2aad15c492324159877587873a4c75d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34527 } }
I20260927 16:39:06.044843 5496 ts_tablet_manager.cc:1397] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 16:39:06.044927 5497 raft_consensus.cc:374] T c0b2189e6edc411695dd7a5f3742c90d P 2aad15c492324159877587873a4c75d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:06.044987 5497 raft_consensus.cc:729] T c0b2189e6edc411695dd7a5f3742c90d P 2aad15c492324159877587873a4c75d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aad15c492324159877587873a4c75d5, State: Initialized, Role: FOLLOWER
I20260927 16:39:06.045372 5497 consensus_queue.cc:260] T c0b2189e6edc411695dd7a5f3742c90d P 2aad15c492324159877587873a4c75d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745f1209d02c45adb1f00ab810e9a40e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43539 } } peers { permanent_uuid: "3ba42588b4354d13991e7f21ba75df3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34575 } } peers { permanent_uuid: "2aad15c492324159877587873a4c75d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34527 } }
I20260927 16:39:06.045292 5498 raft_consensus.cc:348] T c0b2189e6edc411695dd7a5f3742c90d P 3ba42588b4354d13991e7f21ba75df3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745f1209d02c45adb1f00ab810e9a40e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43539 } } peers { permanent_uuid: "3ba42588b4354d13991e7f21ba75df3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34575 } } peers { permanent_uuid: "2aad15c492324159877587873a4c75d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34527 } }
I20260927 16:39:06.045555 5498 raft_consensus.cc:374] T c0b2189e6edc411695dd7a5f3742c90d P 3ba42588b4354d13991e7f21ba75df3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:06.045625 5498 raft_consensus.cc:729] T c0b2189e6edc411695dd7a5f3742c90d P 3ba42588b4354d13991e7f21ba75df3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba42588b4354d13991e7f21ba75df3f, State: Initialized, Role: FOLLOWER
I20260927 16:39:06.046015 5348 heartbeater.cc:503] Master 127.0.98.62:46733 was elected leader, sending a full tablet report...
I20260927 16:39:06.046053 5498 consensus_queue.cc:260] T c0b2189e6edc411695dd7a5f3742c90d P 3ba42588b4354d13991e7f21ba75df3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745f1209d02c45adb1f00ab810e9a40e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43539 } } peers { permanent_uuid: "3ba42588b4354d13991e7f21ba75df3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34575 } } peers { permanent_uuid: "2aad15c492324159877587873a4c75d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34527 } }
I20260927 16:39:06.046330 5497 ts_tablet_manager.cc:1428] T c0b2189e6edc411695dd7a5f3742c90d P 2aad15c492324159877587873a4c75d5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 16:39:06.046933 5498 ts_tablet_manager.cc:1428] T c0b2189e6edc411695dd7a5f3742c90d P 3ba42588b4354d13991e7f21ba75df3f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 16:39:06.047252 5217 heartbeater.cc:503] Master 127.0.98.62:46733 was elected leader, sending a full tablet report...
I20260927 16:39:06.048928 5496 raft_consensus.cc:348] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745f1209d02c45adb1f00ab810e9a40e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43539 } } peers { permanent_uuid: "3ba42588b4354d13991e7f21ba75df3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34575 } } peers { permanent_uuid: "2aad15c492324159877587873a4c75d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34527 } }
I20260927 16:39:06.049134 5496 raft_consensus.cc:374] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:06.049180 5496 raft_consensus.cc:729] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745f1209d02c45adb1f00ab810e9a40e, State: Initialized, Role: FOLLOWER
I20260927 16:39:06.049592 5496 consensus_queue.cc:260] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745f1209d02c45adb1f00ab810e9a40e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43539 } } peers { permanent_uuid: "3ba42588b4354d13991e7f21ba75df3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34575 } } peers { permanent_uuid: "2aad15c492324159877587873a4c75d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34527 } }
I20260927 16:39:06.050163 5479 heartbeater.cc:503] Master 127.0.98.62:46733 was elected leader, sending a full tablet report...
I20260927 16:39:06.050472 5496 ts_tablet_manager.cc:1428] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 16:39:06.066277 5349 tablet.cc:2369] T c0b2189e6edc411695dd7a5f3742c90d P 2aad15c492324159877587873a4c75d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:39:06.147744 5218 tablet.cc:2369] T c0b2189e6edc411695dd7a5f3742c90d P 3ba42588b4354d13991e7f21ba75df3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:39:06.237512 5480 tablet.cc:2369] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:39:06.380611 5504 raft_consensus.cc:482] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:39:06.380851 5504 raft_consensus.cc:504] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745f1209d02c45adb1f00ab810e9a40e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43539 } } peers { permanent_uuid: "3ba42588b4354d13991e7f21ba75df3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34575 } } peers { permanent_uuid: "2aad15c492324159877587873a4c75d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34527 } }
I20260927 16:39:06.382088 5504 leader_election.cc:288] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ba42588b4354d13991e7f21ba75df3f (127.0.98.1:34575), 2aad15c492324159877587873a4c75d5 (127.0.98.2:34527)
I20260927 16:39:06.384330 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:06.382298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34527 (local address 127.0.98.3:50575)
0927 16:39:06.382453 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:06.382469 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:06.382572 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:06.382989 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:06.382992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:06.383010 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:06.383640 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:39:06.383641 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:06.383649 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:39:06.383672 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:06.384125 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:06.384128 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:06.384134 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 16:39:06.384203 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 16:39:06.384225 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 16:39:06.384362 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:06.382298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34575 (local address 127.0.98.3:57645)
0927 16:39:06.382983 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:06.382998 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:06.383094 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:06.383357 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:06.383361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:06.383376 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:06.383813 (+ 437us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:39:06.383814 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:06.383820 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:39:06.383836 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:06.384071 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:06.384074 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:06.384077 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 16:39:06.384158 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 16:39:06.384199 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"mutex_wait_us":336,"thread_start_us":114,"threads_started":1}
I20260927 16:39:06.384364 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:06.382526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57645 (local address 127.0.98.1:34575)
0927 16:39:06.382671 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:06.382675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:06.383085 (+ 410us) server_negotiation.cc:407] Connection header received
0927 16:39:06.383170 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:06.383175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:06.383238 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:06.383312 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:06.383950 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:06.383953 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:06.383955 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:39:06.383965 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:06.383979 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:39:06.383986 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:06.384075 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:06.384192 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 16:39:06.384240 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:39:06.384371 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:06.382577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50575 (local address 127.0.98.2:34527)
0927 16:39:06.382750 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:06.382753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:06.382764 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:39:06.382822 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:06.382825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:06.382861 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:06.382941 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:06.383792 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:06.383796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:06.383798 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:39:06.383809 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:06.383828 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:39:06.383836 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:06.383926 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:06.384231 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 16:39:06.384267 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 16:39:06.384944 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b2189e6edc411695dd7a5f3742c90d" candidate_uuid: "745f1209d02c45adb1f00ab810e9a40e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba42588b4354d13991e7f21ba75df3f" is_pre_election: true
I20260927 16:39:06.385004 5302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b2189e6edc411695dd7a5f3742c90d" candidate_uuid: "745f1209d02c45adb1f00ab810e9a40e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aad15c492324159877587873a4c75d5" is_pre_election: true
I20260927 16:39:06.385262 5170 raft_consensus.cc:2454] T c0b2189e6edc411695dd7a5f3742c90d P 3ba42588b4354d13991e7f21ba75df3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745f1209d02c45adb1f00ab810e9a40e in term 0.
I20260927 16:39:06.385277 5302 raft_consensus.cc:2454] T c0b2189e6edc411695dd7a5f3742c90d P 2aad15c492324159877587873a4c75d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745f1209d02c45adb1f00ab810e9a40e in term 0.
I20260927 16:39:06.385730 5369 leader_election.cc:302] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba42588b4354d13991e7f21ba75df3f, 745f1209d02c45adb1f00ab810e9a40e; no voters:
I20260927 16:39:06.385977 5504 raft_consensus.cc:2781] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:39:06.386057 5504 raft_consensus.cc:482] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:39:06.386116 5504 raft_consensus.cc:3037] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:06.405242 5504 raft_consensus.cc:504] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745f1209d02c45adb1f00ab810e9a40e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43539 } } peers { permanent_uuid: "3ba42588b4354d13991e7f21ba75df3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34575 } } peers { permanent_uuid: "2aad15c492324159877587873a4c75d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34527 } }
I20260927 16:39:06.405956 5302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b2189e6edc411695dd7a5f3742c90d" candidate_uuid: "745f1209d02c45adb1f00ab810e9a40e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aad15c492324159877587873a4c75d5"
I20260927 16:39:06.405979 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b2189e6edc411695dd7a5f3742c90d" candidate_uuid: "745f1209d02c45adb1f00ab810e9a40e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba42588b4354d13991e7f21ba75df3f"
I20260927 16:39:06.406090 5302 raft_consensus.cc:3037] T c0b2189e6edc411695dd7a5f3742c90d P 2aad15c492324159877587873a4c75d5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:06.406104 5170 raft_consensus.cc:3037] T c0b2189e6edc411695dd7a5f3742c90d P 3ba42588b4354d13991e7f21ba75df3f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:06.407382 5302 raft_consensus.cc:2454] T c0b2189e6edc411695dd7a5f3742c90d P 2aad15c492324159877587873a4c75d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745f1209d02c45adb1f00ab810e9a40e in term 1.
I20260927 16:39:06.407696 5366 leader_election.cc:302] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aad15c492324159877587873a4c75d5, 745f1209d02c45adb1f00ab810e9a40e; no voters:
I20260927 16:39:06.407766 5170 raft_consensus.cc:2454] T c0b2189e6edc411695dd7a5f3742c90d P 3ba42588b4354d13991e7f21ba75df3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745f1209d02c45adb1f00ab810e9a40e in term 1.
I20260927 16:39:06.407922 5504 leader_election.cc:288] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e [CANDIDATE]: Term 1 election: Requested vote from peers 3ba42588b4354d13991e7f21ba75df3f (127.0.98.1:34575), 2aad15c492324159877587873a4c75d5 (127.0.98.2:34527)
I20260927 16:39:06.407999 5504 raft_consensus.cc:2781] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:39:06.408059 5504 raft_consensus.cc:686] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e [term 1 LEADER]: Becoming Leader. State: Replica: 745f1209d02c45adb1f00ab810e9a40e, State: Running, Role: LEADER
I20260927 16:39:06.408386 5504 consensus_queue.cc:237] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745f1209d02c45adb1f00ab810e9a40e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43539 } } peers { permanent_uuid: "3ba42588b4354d13991e7f21ba75df3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34575 } } peers { permanent_uuid: "2aad15c492324159877587873a4c75d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34527 } }
I20260927 16:39:06.412123 5028 catalog_manager.cc:5495] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e reported cstate change: term changed from 0 to 1, leader changed from <none> to 745f1209d02c45adb1f00ab810e9a40e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "745f1209d02c45adb1f00ab810e9a40e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745f1209d02c45adb1f00ab810e9a40e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ba42588b4354d13991e7f21ba75df3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aad15c492324159877587873a4c75d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34527 } health_report { overall_health: UNKNOWN } } }
I20260927 16:39:06.477401 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:06.476031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53164 (local address 127.0.98.3:43539)
0927 16:39:06.476155 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:06.476158 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:06.476170 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:39:06.476241 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:06.476243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:06.476297 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:06.476366 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:06.477034 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:06.477036 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:06.477038 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:39:06.477045 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:06.477058 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:39:06.477063 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:06.477126 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:06.477204 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 16:39:06.477234 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:39:06.482952 5302 raft_consensus.cc:1261] T c0b2189e6edc411695dd7a5f3742c90d P 2aad15c492324159877587873a4c75d5 [term 1 FOLLOWER]: Refusing update from remote peer 745f1209d02c45adb1f00ab810e9a40e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:39:06.483009 5170 raft_consensus.cc:1261] T c0b2189e6edc411695dd7a5f3742c90d P 3ba42588b4354d13991e7f21ba75df3f [term 1 FOLLOWER]: Refusing update from remote peer 745f1209d02c45adb1f00ab810e9a40e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:39:06.483696 5504 consensus_queue.cc:1035] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aad15c492324159877587873a4c75d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:39:06.483920 5506 consensus_queue.cc:1035] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba42588b4354d13991e7f21ba75df3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:39:06.490669 5511 mvcc.cc:205] Tried to move back new op lower bound from 7333999191987351552 to 7333999191692709888. Current Snapshot: MvccSnapshot[applied={T|T < 7333999191987351552}]
I20260927 16:39:06.497355 5510 mvcc.cc:205] Tried to move back new op lower bound from 7333999191987351552 to 7333999191692709888. Current Snapshot: MvccSnapshot[applied={T|T < 7333999191987351552 or (T in {7333999191987351552})}]
I20260927 16:39:06.498081 5028 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:39:06.498257 5028 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:39:06.500492 5028 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4ab2c62a94f40aa90ab5aa9a0164fc4: Sending DeleteTablet for 3 replicas of tablet c0b2189e6edc411695dd7a5f3742c90d
I20260927 16:39:06.501153 5151 tablet_service.cc:1514] Processing DeleteTablet for tablet c0b2189e6edc411695dd7a5f3742c90d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:39:06 UTC) from {username='slave'} at 127.0.0.1:58678
I20260927 16:39:06.501158 5400 tablet_service.cc:1514] Processing DeleteTablet for tablet c0b2189e6edc411695dd7a5f3742c90d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:39:06 UTC) from {username='slave'} at 127.0.0.1:53160
I20260927 16:39:06.501461 5282 tablet_service.cc:1514] Processing DeleteTablet for tablet c0b2189e6edc411695dd7a5f3742c90d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:39:06 UTC) from {username='slave'} at 127.0.0.1:41030
I20260927 16:39:06.501585 5519 tablet_replica.cc:330] T c0b2189e6edc411695dd7a5f3742c90d P 3ba42588b4354d13991e7f21ba75df3f: stopping tablet replica
I20260927 16:39:06.501799 5520 tablet_replica.cc:330] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e: stopping tablet replica
I20260927 16:39:06.501883 5519 raft_consensus.cc:2229] T c0b2189e6edc411695dd7a5f3742c90d P 3ba42588b4354d13991e7f21ba75df3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:39:06.502162 5519 raft_consensus.cc:2258] T c0b2189e6edc411695dd7a5f3742c90d P 3ba42588b4354d13991e7f21ba75df3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:39:06.502202 5520 raft_consensus.cc:2229] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:39:06.502457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 5089
I20260927 16:39:06.502642 5520 raft_consensus.cc:2258] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:39:06.504210 5520 ts_tablet_manager.cc:1905] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:39:06.507036 5520 ts_tablet_manager.cc:1918] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:39:06.507191 5520 log.cc:1196] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790527121515808-392-0/minicluster-data/ts-2/wal/wals/c0b2189e6edc411695dd7a5f3742c90d
I20260927 16:39:06.507608 5520 ts_tablet_manager.cc:1939] T c0b2189e6edc411695dd7a5f3742c90d P 745f1209d02c45adb1f00ab810e9a40e: Deleting consensus metadata
W20260927 16:39:06.511133 5016 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34575 (error 108)
I20260927 16:39:06.511313 5521 tablet_replica.cc:330] T c0b2189e6edc411695dd7a5f3742c90d P 2aad15c492324159877587873a4c75d5: stopping tablet replica
I20260927 16:39:06.511596 5521 raft_consensus.cc:2229] T c0b2189e6edc411695dd7a5f3742c90d P 2aad15c492324159877587873a4c75d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:39:06.511888 5521 raft_consensus.cc:2258] T c0b2189e6edc411695dd7a5f3742c90d P 2aad15c492324159877587873a4c75d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:39:06.512369 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 5220
I20260927 16:39:06.512442 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:06.512167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34575 (local address 127.0.0.1:58686)
0927 16:39:06.512276 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:06.512355 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 16:39:06.513098 5016 catalog_manager.cc:4584] TS 3ba42588b4354d13991e7f21ba75df3f (127.0.98.1:34575): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c0b2189e6edc411695dd7a5f3742c90d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34575: connect: Connection refused (error 111)
I20260927 16:39:06.518203 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 5351
I20260927 16:39:06.518342 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:06.518096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34527 (local address 127.0.0.1:41046)
0927 16:39:06.518199 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:06.518248 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 16:39:06.544520 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 4998
2026-09-27T16:39:06Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1322 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 16:39:06.565354 392 mini_oidc.cc:152] Starting JWKS server
I20260927 16:39:06.566962 392 webserver.cc:460] Webserver started at https://127.0.0.1:43507/ using document root <none> and password file <none>
I20260927 16:39:06.567088 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 16:39:06.567399 392 webserver.cc:460] Webserver started at http://127.0.0.1:41371/ using document root <none> and password file <none>
2026-09-27T16:39:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:39:06Z Disabled control of system clock
I20260927 16:39:06.576503 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42843
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42843
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43507/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790527121515808-392-0/testchainca.cert with env {}
W20260927 16:39:06.683358 5528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:06.683583 5528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:06.683619 5528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:06.687493 5528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:39:06.687534 5528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:06.687547 5528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:06.687562 5528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:39:06.687575 5528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 16:39:06.687588 5528 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 16:39:06.687600 5528 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43507/jwks.json
I20260927 16:39:06.692301 5528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42843
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43507/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790527121515808-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:06.693089 5528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:06.693944 5528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:06.701659 5533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:06.701618 5536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:06.701628 5534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:06.701777 5528 server_base.cc:1034] running on GCE node
I20260927 16:39:06.702437 5528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:06.703645 5528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:06.704797 5528 hybrid_clock.cc:648] HybridClock initialized: now 1790527146704779 us; error 28 us; skew 500 ppm
I20260927 16:39:06.706353 5528 webserver.cc:460] Webserver started at http://127.0.98.62:44165/ using document root <none> and password file <none>
I20260927 16:39:06.706830 5528 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:06.706882 5528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:06.707017 5528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:06.708829 5528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "cc08b45202ac4302a2adc426dbd2f092"
format_stamp: "Formatted at 2026-09-27 16:39:06 on dist-test-slave-z40b"
server_key: "47bdc5e9ee3e528b008fcb9c373929df"
server_key_iv: "5160917aa57155eb689bb0ea56c76ede"
server_key_version: "encryptionkey@0"
I20260927 16:39:06.709287 5528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "cc08b45202ac4302a2adc426dbd2f092"
format_stamp: "Formatted at 2026-09-27 16:39:06 on dist-test-slave-z40b"
server_key: "47bdc5e9ee3e528b008fcb9c373929df"
server_key_iv: "5160917aa57155eb689bb0ea56c76ede"
server_key_version: "encryptionkey@0"
I20260927 16:39:06.712767 5528 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 16:39:06.715097 5542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:06.716291 5528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:39:06.716434 5528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "cc08b45202ac4302a2adc426dbd2f092"
format_stamp: "Formatted at 2026-09-27 16:39:06 on dist-test-slave-z40b"
server_key: "47bdc5e9ee3e528b008fcb9c373929df"
server_key_iv: "5160917aa57155eb689bb0ea56c76ede"
server_key_version: "encryptionkey@0"
I20260927 16:39:06.716532 5528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:06.726871 5528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:06.727420 5528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:06.727582 5528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:06.733559 5528 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42843
I20260927 16:39:06.733572 5594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42843 every 8 connection(s)
I20260927 16:39:06.734413 5528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:39:06.737423 5595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:06.741406 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 5528
I20260927 16:39:06.741557 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:39:06.741818 392 external_mini_cluster.cc:1428] Setting key 6d97efc3c41478a12aa5e1b61d1303f5
I20260927 16:39:06.743041 5595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc08b45202ac4302a2adc426dbd2f092: Bootstrap starting.
I20260927 16:39:06.745275 5595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc08b45202ac4302a2adc426dbd2f092: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:06.745987 5595 log.cc:824] T 00000000000000000000000000000000 P cc08b45202ac4302a2adc426dbd2f092: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:06.747743 5595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc08b45202ac4302a2adc426dbd2f092: No bootstrap required, opened a new log
I20260927 16:39:06.750613 5598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:06.743041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59496 (local address 127.0.98.62:42843)
0927 16:39:06.743693 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:06.743706 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:06.743739 (+ 33us) server_negotiation.cc:407] Connection header received
0927 16:39:06.744336 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:06.744352 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:06.744708 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:06.745034 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:06.745538 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:06.746524 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:06.747495 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:06.747992 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:06.748166 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:06.748978 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:06.748982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:06.748993 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:39:06.749387 (+ 394us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:06.749454 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:39:06.749473 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:06.749580 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:06.749731 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 16:39:06.749959 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":149,"threads_started":1}
I20260927 16:39:06.751461 5595 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc08b45202ac4302a2adc426dbd2f092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc08b45202ac4302a2adc426dbd2f092" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42843 } }
I20260927 16:39:06.751682 5595 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc08b45202ac4302a2adc426dbd2f092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:06.751763 5595 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc08b45202ac4302a2adc426dbd2f092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc08b45202ac4302a2adc426dbd2f092, State: Initialized, Role: FOLLOWER
I20260927 16:39:06.752305 5595 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc08b45202ac4302a2adc426dbd2f092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc08b45202ac4302a2adc426dbd2f092" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42843 } }
I20260927 16:39:06.752476 5595 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc08b45202ac4302a2adc426dbd2f092 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:39:06.752537 5595 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc08b45202ac4302a2adc426dbd2f092 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:39:06.752643 5595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc08b45202ac4302a2adc426dbd2f092 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:06.753278 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:39:06.786370 5595 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc08b45202ac4302a2adc426dbd2f092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc08b45202ac4302a2adc426dbd2f092" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42843 } }
I20260927 16:39:06.786849 5595 leader_election.cc:302] T 00000000000000000000000000000000 P cc08b45202ac4302a2adc426dbd2f092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc08b45202ac4302a2adc426dbd2f092; no voters:
I20260927 16:39:06.787194 5595 leader_election.cc:288] T 00000000000000000000000000000000 P cc08b45202ac4302a2adc426dbd2f092 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:39:06.787348 5600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc08b45202ac4302a2adc426dbd2f092 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:39:06.787595 5600 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc08b45202ac4302a2adc426dbd2f092 [term 1 LEADER]: Becoming Leader. State: Replica: cc08b45202ac4302a2adc426dbd2f092, State: Running, Role: LEADER
I20260927 16:39:06.787915 5600 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc08b45202ac4302a2adc426dbd2f092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc08b45202ac4302a2adc426dbd2f092" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42843 } }
I20260927 16:39:06.788636 5595 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc08b45202ac4302a2adc426dbd2f092 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:39:06.789016 5602 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc08b45202ac4302a2adc426dbd2f092 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc08b45202ac4302a2adc426dbd2f092. Latest consensus state: current_term: 1 leader_uuid: "cc08b45202ac4302a2adc426dbd2f092" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc08b45202ac4302a2adc426dbd2f092" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42843 } } }
I20260927 16:39:06.789150 5602 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc08b45202ac4302a2adc426dbd2f092 [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:06.789383 5601 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc08b45202ac4302a2adc426dbd2f092 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc08b45202ac4302a2adc426dbd2f092" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc08b45202ac4302a2adc426dbd2f092" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42843 } } }
I20260927 16:39:06.789481 5601 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc08b45202ac4302a2adc426dbd2f092 [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:06.789803 5604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:39:06.793038 5604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:39:06.798362 5604 catalog_manager.cc:1347] Generated new cluster ID: 1ac8409f1b834416b91f1fb0c85f82d6
I20260927 16:39:06.798419 5604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:39:06.827347 5604 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:39:06.828608 5604 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:39:06.835553 5604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc08b45202ac4302a2adc426dbd2f092: Generated new TSK 0
I20260927 16:39:06.836143 5604 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 16:39:06.864040 5625 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 16:39:06.864148 5625 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 16:39:06.864656 5598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:06.848011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59510 (local address 127.0.98.62:42843)
0927 16:39:06.848141 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:06.848146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:06.848316 (+ 170us) server_negotiation.cc:407] Connection header received
0927 16:39:06.848486 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:06.848489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:06.848546 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:06.848664 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0927 16:39:06.848882 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:06.849693 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:06.850501 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:06.850843 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:06.850952 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:06.851244 (+ 292us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 16:39:06.864306 (+ 13062us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43507/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 16:39:06.864527 (+ 221us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59510: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43507/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 16:39:06.864727 5598 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59510: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43507/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 16:39:06.865737 5624 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:06.847909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42843 (local address 127.0.0.1:59510)
0927 16:39:06.848237 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:06.848250 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:06.848348 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:06.848678 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:06.848680 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:06.848686 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0927 16:39:06.848788 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:06.848792 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:06.849801 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:06.849803 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:06.850400 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:06.850406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:06.850952 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:06.850967 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:06.851027 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:06.851062 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 16:39:06.864481 (+ 13419us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43507/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 16:39:06.864641 (+ 160us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42843: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43507/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":96,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:42843: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43507/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 16:39:06.866410 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 5528
2026-09-27T16:39:06Z chronyd exiting
I20260927 16:39:06.903724 392 test_util.cc:175] -----------------------------------------------
I20260927 16:39:06.903792 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790527121515808-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (339 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 16:39:06.904750 392 mini_oidc.cc:152] Starting JWKS server
I20260927 16:39:06.905588 392 webserver.cc:460] Webserver started at https://127.0.0.1:43145/ using document root <none> and password file <none>
I20260927 16:39:06.905700 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 16:39:06.905990 392 webserver.cc:460] Webserver started at http://127.0.0.1:42203/ using document root <none> and password file <none>
2026-09-27T16:39:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:39:06Z Disabled control of system clock
I20260927 16:39:06.914215 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37285
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37285
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43145/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790527121515808-392-0/testchainca.cert with env {}
W20260927 16:39:07.014946 5631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:07.015202 5631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:07.015290 5631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:07.018862 5631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:39:07.018937 5631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:07.018965 5631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:07.019001 5631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:39:07.019032 5631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 16:39:07.019064 5631 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 16:39:07.019090 5631 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43145/jwks.json
I20260927 16:39:07.023916 5631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37285
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43145/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790527121515808-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:07.024796 5631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:07.025709 5631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:07.033635 5636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:07.033643 5637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:07.033671 5639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:07.033728 5631 server_base.cc:1034] running on GCE node
I20260927 16:39:07.034379 5631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:07.035382 5631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:07.036551 5631 hybrid_clock.cc:648] HybridClock initialized: now 1790527147036530 us; error 33 us; skew 500 ppm
I20260927 16:39:07.038218 5631 webserver.cc:460] Webserver started at http://127.0.98.62:41973/ using document root <none> and password file <none>
I20260927 16:39:07.038731 5631 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:07.038813 5631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:07.038993 5631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:07.040932 5631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "89285936347e4cffa7e4bd89c277bd33"
format_stamp: "Formatted at 2026-09-27 16:39:07 on dist-test-slave-z40b"
server_key: "3034f8172605c0f59a51c9bab1210780"
server_key_iv: "310b9436ee5a08252edd0cc7e7c17b4f"
server_key_version: "encryptionkey@0"
I20260927 16:39:07.041471 5631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "89285936347e4cffa7e4bd89c277bd33"
format_stamp: "Formatted at 2026-09-27 16:39:07 on dist-test-slave-z40b"
server_key: "3034f8172605c0f59a51c9bab1210780"
server_key_iv: "310b9436ee5a08252edd0cc7e7c17b4f"
server_key_version: "encryptionkey@0"
I20260927 16:39:07.044771 5631 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 16:39:07.046901 5645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:07.048008 5631 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:07.048166 5631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "89285936347e4cffa7e4bd89c277bd33"
format_stamp: "Formatted at 2026-09-27 16:39:07 on dist-test-slave-z40b"
server_key: "3034f8172605c0f59a51c9bab1210780"
server_key_iv: "310b9436ee5a08252edd0cc7e7c17b4f"
server_key_version: "encryptionkey@0"
I20260927 16:39:07.048273 5631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:07.064409 5631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:07.064981 5631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:07.065196 5631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:07.071588 5631 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37285
I20260927 16:39:07.071641 5697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37285 every 8 connection(s)
I20260927 16:39:07.072456 5631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:39:07.075055 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:07.079538 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 5631
I20260927 16:39:07.079627 5698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89285936347e4cffa7e4bd89c277bd33: Bootstrap starting.
I20260927 16:39:07.079689 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:39:07.079957 392 external_mini_cluster.cc:1428] Setting key 1a1ed23d0c2feadfb07be3909b0b2daa
I20260927 16:39:07.081969 5698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89285936347e4cffa7e4bd89c277bd33: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:07.082623 5698 log.cc:824] T 00000000000000000000000000000000 P 89285936347e4cffa7e4bd89c277bd33: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:07.084288 5698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89285936347e4cffa7e4bd89c277bd33: No bootstrap required, opened a new log
I20260927 16:39:07.087553 5698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89285936347e4cffa7e4bd89c277bd33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89285936347e4cffa7e4bd89c277bd33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37285 } }
I20260927 16:39:07.087785 5698 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89285936347e4cffa7e4bd89c277bd33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:07.087867 5698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89285936347e4cffa7e4bd89c277bd33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89285936347e4cffa7e4bd89c277bd33, State: Initialized, Role: FOLLOWER
I20260927 16:39:07.088234 5701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:07.081111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50954 (local address 127.0.98.62:37285)
0927 16:39:07.081666 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:07.081679 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:07.081710 (+ 31us) server_negotiation.cc:407] Connection header received
0927 16:39:07.082299 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:07.082315 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:07.082651 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:07.082994 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:07.083417 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:07.084292 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:07.085138 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:07.085605 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:07.085774 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:07.086578 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:07.086583 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:07.086593 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:39:07.086938 (+ 345us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:07.086985 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:39:07.087003 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:07.087112 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:07.087236 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 16:39:07.087463 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":115,"threads_started":1}
I20260927 16:39:07.088328 5698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89285936347e4cffa7e4bd89c277bd33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89285936347e4cffa7e4bd89c277bd33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37285 } }
I20260927 16:39:07.088495 5698 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89285936347e4cffa7e4bd89c277bd33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:39:07.088562 5698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89285936347e4cffa7e4bd89c277bd33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:39:07.088652 5698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89285936347e4cffa7e4bd89c277bd33 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:07.089530 5698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89285936347e4cffa7e4bd89c277bd33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89285936347e4cffa7e4bd89c277bd33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37285 } }
I20260927 16:39:07.089861 5698 leader_election.cc:302] T 00000000000000000000000000000000 P 89285936347e4cffa7e4bd89c277bd33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89285936347e4cffa7e4bd89c277bd33; no voters:
I20260927 16:39:07.090193 5698 leader_election.cc:288] T 00000000000000000000000000000000 P 89285936347e4cffa7e4bd89c277bd33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:39:07.090276 5703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89285936347e4cffa7e4bd89c277bd33 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:39:07.090418 5703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89285936347e4cffa7e4bd89c277bd33 [term 1 LEADER]: Becoming Leader. State: Replica: 89285936347e4cffa7e4bd89c277bd33, State: Running, Role: LEADER
I20260927 16:39:07.090709 5703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89285936347e4cffa7e4bd89c277bd33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89285936347e4cffa7e4bd89c277bd33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37285 } }
I20260927 16:39:07.090848 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:39:07.091310 5698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89285936347e4cffa7e4bd89c277bd33 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:39:07.092813 5705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89285936347e4cffa7e4bd89c277bd33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89285936347e4cffa7e4bd89c277bd33. Latest consensus state: current_term: 1 leader_uuid: "89285936347e4cffa7e4bd89c277bd33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89285936347e4cffa7e4bd89c277bd33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37285 } } }
I20260927 16:39:07.092905 5705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89285936347e4cffa7e4bd89c277bd33 [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:07.091758 5704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89285936347e4cffa7e4bd89c277bd33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89285936347e4cffa7e4bd89c277bd33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89285936347e4cffa7e4bd89c277bd33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37285 } } }
I20260927 16:39:07.093132 5704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89285936347e4cffa7e4bd89c277bd33 [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:07.093358 5717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:39:07.095860 5717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:39:07.101084 5717 catalog_manager.cc:1347] Generated new cluster ID: 94f8eefdae664e22b4d55ebd69d1a2bb
I20260927 16:39:07.101155 5717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:39:07.113315 5717 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:39:07.114120 5717 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:39:07.128270 5717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89285936347e4cffa7e4bd89c277bd33: Generated new TSK 0
I20260927 16:39:07.128854 5717 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 16:39:07.162106 5728 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 16:39:07.162182 5728 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 16:39:07.162698 5701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:07.140703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.0.98.62:37285)
0927 16:39:07.140822 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:07.140828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:07.140974 (+ 146us) server_negotiation.cc:407] Connection header received
0927 16:39:07.141120 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:07.141122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:07.141162 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:07.141242 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0927 16:39:07.141484 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:07.141982 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:07.143132 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:07.143387 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:07.143456 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:07.143726 (+ 270us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 16:39:07.162387 (+ 18661us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 16:39:07.162567 (+ 180us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50962: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 16:39:07.162770 5701 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50962: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 16:39:07.162865 5727 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:07.140610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37285 (local address 127.0.0.1:50962)
0927 16:39:07.140890 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:07.140903 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:07.141009 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:07.141264 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:07.141267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:07.141274 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0927 16:39:07.141385 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:07.141391 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:07.142085 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:07.142088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:07.143028 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:07.143037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:07.143470 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:07.143487 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:07.143571 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:07.143597 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 16:39:07.162531 (+ 18934us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 16:39:07.162694 (+ 163us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37285: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":105,"threads_started":1}
I20260927 16:39:07.163535 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 5631
2026-09-27T16:39:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (296 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 16:39:07.202153 392 mini_oidc.cc:152] Starting JWKS server
I20260927 16:39:07.202975 392 webserver.cc:460] Webserver started at https://127.0.0.1:42695/ using document root <none> and password file <none>
I20260927 16:39:07.203100 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 16:39:07.203384 392 webserver.cc:460] Webserver started at http://127.0.0.1:42131/ using document root <none> and password file <none>
2026-09-27T16:39:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:39:07Z Disabled control of system clock
I20260927 16:39:07.211354 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40175
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40175
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42695/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 16:39:07.313217 5734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:07.313488 5734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:07.313572 5734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:07.317356 5734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:39:07.317440 5734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:07.317468 5734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:07.317504 5734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:39:07.317543 5734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 16:39:07.317569 5734 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 16:39:07.317600 5734 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42695/jwks.json
I20260927 16:39:07.322084 5734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40175
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42695/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:07.323026 5734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:07.324278 5734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:07.332219 5739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:07.332229 5740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:07.332307 5742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:07.332481 5734 server_base.cc:1034] running on GCE node
I20260927 16:39:07.333078 5734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:07.334070 5734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:07.335266 5734 hybrid_clock.cc:648] HybridClock initialized: now 1790527147335222 us; error 48 us; skew 500 ppm
I20260927 16:39:07.336900 5734 webserver.cc:460] Webserver started at http://127.0.98.62:35133/ using document root <none> and password file <none>
I20260927 16:39:07.337421 5734 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:07.337478 5734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:07.337718 5734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:07.339643 5734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "97e6985bb48e4655ad86e49502bd85cc"
format_stamp: "Formatted at 2026-09-27 16:39:07 on dist-test-slave-z40b"
server_key: "a59e86d136977654a3d32812827ce601"
server_key_iv: "87a7b4f73a45f473dc01ab914d127bd8"
server_key_version: "encryptionkey@0"
I20260927 16:39:07.340171 5734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "97e6985bb48e4655ad86e49502bd85cc"
format_stamp: "Formatted at 2026-09-27 16:39:07 on dist-test-slave-z40b"
server_key: "a59e86d136977654a3d32812827ce601"
server_key_iv: "87a7b4f73a45f473dc01ab914d127bd8"
server_key_version: "encryptionkey@0"
I20260927 16:39:07.343469 5734 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:39:07.345566 5748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:07.346696 5734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:07.346834 5734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "97e6985bb48e4655ad86e49502bd85cc"
format_stamp: "Formatted at 2026-09-27 16:39:07 on dist-test-slave-z40b"
server_key: "a59e86d136977654a3d32812827ce601"
server_key_iv: "87a7b4f73a45f473dc01ab914d127bd8"
server_key_version: "encryptionkey@0"
I20260927 16:39:07.346935 5734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:07.362363 5734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:07.362937 5734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:07.363116 5734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:07.369452 5734 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40175
I20260927 16:39:07.369467 5800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40175 every 8 connection(s)
I20260927 16:39:07.370296 5734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:39:07.373235 5801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:07.376282 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 5734
I20260927 16:39:07.376402 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:39:07.376690 392 external_mini_cluster.cc:1428] Setting key 8fb4acfb1cbd5c7e89f90238a856cc2b
I20260927 16:39:07.378711 5801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97e6985bb48e4655ad86e49502bd85cc: Bootstrap starting.
I20260927 16:39:07.380841 5801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97e6985bb48e4655ad86e49502bd85cc: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:07.381528 5801 log.cc:824] T 00000000000000000000000000000000 P 97e6985bb48e4655ad86e49502bd85cc: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:07.383301 5801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97e6985bb48e4655ad86e49502bd85cc: No bootstrap required, opened a new log
I20260927 16:39:07.385653 5804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:07.377942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41248 (local address 127.0.98.62:40175)
0927 16:39:07.378526 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:07.378542 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:07.378574 (+ 32us) server_negotiation.cc:407] Connection header received
0927 16:39:07.379271 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:07.379288 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:07.379660 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:07.379971 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:07.380377 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:07.381399 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:07.382518 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:07.382953 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:07.383134 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:07.383942 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:07.383947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:07.383958 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:39:07.384310 (+ 352us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:07.384359 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:39:07.384377 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:07.384483 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:07.384615 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 16:39:07.384853 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":129,"threads_started":1}
I20260927 16:39:07.386394 5801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97e6985bb48e4655ad86e49502bd85cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97e6985bb48e4655ad86e49502bd85cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40175 } }
I20260927 16:39:07.386649 5801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97e6985bb48e4655ad86e49502bd85cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:07.386734 5801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97e6985bb48e4655ad86e49502bd85cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97e6985bb48e4655ad86e49502bd85cc, State: Initialized, Role: FOLLOWER
I20260927 16:39:07.387197 5801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97e6985bb48e4655ad86e49502bd85cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97e6985bb48e4655ad86e49502bd85cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40175 } }
I20260927 16:39:07.387401 5801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97e6985bb48e4655ad86e49502bd85cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:39:07.387470 5801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97e6985bb48e4655ad86e49502bd85cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:39:07.387569 5801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97e6985bb48e4655ad86e49502bd85cc [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:07.388473 5801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97e6985bb48e4655ad86e49502bd85cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97e6985bb48e4655ad86e49502bd85cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40175 } }
I20260927 16:39:07.388514 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:39:07.388801 5801 leader_election.cc:302] T 00000000000000000000000000000000 P 97e6985bb48e4655ad86e49502bd85cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97e6985bb48e4655ad86e49502bd85cc; no voters:
I20260927 16:39:07.389086 5801 leader_election.cc:288] T 00000000000000000000000000000000 P 97e6985bb48e4655ad86e49502bd85cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:39:07.389158 5806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97e6985bb48e4655ad86e49502bd85cc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:39:07.389310 5806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97e6985bb48e4655ad86e49502bd85cc [term 1 LEADER]: Becoming Leader. State: Replica: 97e6985bb48e4655ad86e49502bd85cc, State: Running, Role: LEADER
I20260927 16:39:07.389602 5806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97e6985bb48e4655ad86e49502bd85cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97e6985bb48e4655ad86e49502bd85cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40175 } }
I20260927 16:39:07.390228 5801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97e6985bb48e4655ad86e49502bd85cc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:39:07.390713 5806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97e6985bb48e4655ad86e49502bd85cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97e6985bb48e4655ad86e49502bd85cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97e6985bb48e4655ad86e49502bd85cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40175 } } }
I20260927 16:39:07.390825 5806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97e6985bb48e4655ad86e49502bd85cc [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:07.391021 5806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97e6985bb48e4655ad86e49502bd85cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97e6985bb48e4655ad86e49502bd85cc. Latest consensus state: current_term: 1 leader_uuid: "97e6985bb48e4655ad86e49502bd85cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97e6985bb48e4655ad86e49502bd85cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40175 } } }
I20260927 16:39:07.391126 5806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97e6985bb48e4655ad86e49502bd85cc [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:07.391342 5810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:39:07.394075 5810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:39:07.399345 5810 catalog_manager.cc:1347] Generated new cluster ID: 3bcb6f3293da404786d0bc3cea5a8af8
I20260927 16:39:07.399403 5810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:39:07.418435 5810 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:39:07.419498 5810 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:39:07.434769 5810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97e6985bb48e4655ad86e49502bd85cc: Generated new TSK 0
I20260927 16:39:07.435477 5810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:39:07.453147 5804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:07.448243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41254 (local address 127.0.98.62:40175)
0927 16:39:07.448370 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:07.448377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:07.448537 (+ 160us) server_negotiation.cc:407] Connection header received
0927 16:39:07.448677 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:07.448679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:07.448721 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:07.448804 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0927 16:39:07.449032 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:07.449610 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:07.450869 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:07.451070 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:07.451236 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:07.451530 (+ 294us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 16:39:07.452799 (+ 1269us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42695/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 16:39:07.452996 (+ 197us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41254: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42695/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 16:39:07.453202 5804 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41254: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42695/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 16:39:07.455066 5830 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:07.448155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40175 (local address 127.0.0.1:41254)
0927 16:39:07.448455 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:07.448469 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:07.448572 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:07.448825 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:07.448829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:07.448836 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0927 16:39:07.448944 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:07.448950 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:07.449715 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:07.449718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:07.450747 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:07.450757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:07.451235 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:07.451255 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:07.451345 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:07.451372 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 16:39:07.452942 (+ 1570us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42695/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 16:39:07.454913 (+ 1971us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40175: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42695/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":1722,"thread_start_us":95,"threads_started":1}
I20260927 16:39:07.455698 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 5734
E20260927 16:39:07.463255 5831 webserver.cc:533] Webserver: sslize failed:
E20260927 16:39:07.463326 5831 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T16:39:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (292 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T16:39:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:39:07Z Disabled control of system clock
I20260927 16:39:07.502358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35169
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35169
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790527121515808-392-0/testchainca.cert with env {}
W20260927 16:39:07.603627 5835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:07.603849 5835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:07.603912 5835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:07.607735 5835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:39:07.607802 5835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:07.607829 5835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:07.607878 5835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:39:07.607914 5835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 16:39:07.607951 5835 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 16:39:07.607983 5835 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 16:39:07.612820 5835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35169
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790527121515808-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:07.613700 5835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:07.614574 5835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:07.622328 5841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:07.622290 5843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:07.622296 5840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:07.622407 5835 server_base.cc:1034] running on GCE node
I20260927 16:39:07.623071 5835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:07.624017 5835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:07.625200 5835 hybrid_clock.cc:648] HybridClock initialized: now 1790527147625185 us; error 31 us; skew 500 ppm
I20260927 16:39:07.626935 5835 webserver.cc:460] Webserver started at http://127.0.98.62:40707/ using document root <none> and password file <none>
I20260927 16:39:07.627509 5835 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:07.627592 5835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:07.627864 5835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:07.629695 5835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "399fb80d407d4bdc87cd1c9c83685b04"
format_stamp: "Formatted at 2026-09-27 16:39:07 on dist-test-slave-z40b"
server_key: "b6062c1620af8c4255a458c250fc180c"
server_key_iv: "5f016030bec72626664bd5425baaead1"
server_key_version: "encryptionkey@0"
I20260927 16:39:07.630245 5835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "399fb80d407d4bdc87cd1c9c83685b04"
format_stamp: "Formatted at 2026-09-27 16:39:07 on dist-test-slave-z40b"
server_key: "b6062c1620af8c4255a458c250fc180c"
server_key_iv: "5f016030bec72626664bd5425baaead1"
server_key_version: "encryptionkey@0"
I20260927 16:39:07.633392 5835 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 16:39:07.635442 5849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:07.636611 5835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:39:07.636746 5835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "399fb80d407d4bdc87cd1c9c83685b04"
format_stamp: "Formatted at 2026-09-27 16:39:07 on dist-test-slave-z40b"
server_key: "b6062c1620af8c4255a458c250fc180c"
server_key_iv: "5f016030bec72626664bd5425baaead1"
server_key_version: "encryptionkey@0"
I20260927 16:39:07.636847 5835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:07.660575 5835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:07.661162 5835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:07.661345 5835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:07.667464 5835 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35169
I20260927 16:39:07.667472 5901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35169 every 8 connection(s)
I20260927 16:39:07.668325 5835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:39:07.671013 5902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:07.675649 5902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 399fb80d407d4bdc87cd1c9c83685b04: Bootstrap starting.
I20260927 16:39:07.677587 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 5835
I20260927 16:39:07.677742 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:39:07.677968 5902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 399fb80d407d4bdc87cd1c9c83685b04: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:07.678045 392 external_mini_cluster.cc:1428] Setting key 9c2c063c0a85a6687f8e72e87ad63226
I20260927 16:39:07.678786 5902 log.cc:824] T 00000000000000000000000000000000 P 399fb80d407d4bdc87cd1c9c83685b04: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:07.680904 5902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 399fb80d407d4bdc87cd1c9c83685b04: No bootstrap required, opened a new log
I20260927 16:39:07.685022 5902 raft_consensus.cc:348] T 00000000000000000000000000000000 P 399fb80d407d4bdc87cd1c9c83685b04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399fb80d407d4bdc87cd1c9c83685b04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35169 } }
I20260927 16:39:07.685230 5902 raft_consensus.cc:374] T 00000000000000000000000000000000 P 399fb80d407d4bdc87cd1c9c83685b04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:07.685298 5902 raft_consensus.cc:729] T 00000000000000000000000000000000 P 399fb80d407d4bdc87cd1c9c83685b04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 399fb80d407d4bdc87cd1c9c83685b04, State: Initialized, Role: FOLLOWER
I20260927 16:39:07.685807 5902 consensus_queue.cc:260] T 00000000000000000000000000000000 P 399fb80d407d4bdc87cd1c9c83685b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399fb80d407d4bdc87cd1c9c83685b04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35169 } }
I20260927 16:39:07.685967 5902 raft_consensus.cc:388] T 00000000000000000000000000000000 P 399fb80d407d4bdc87cd1c9c83685b04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:39:07.686033 5902 raft_consensus.cc:482] T 00000000000000000000000000000000 P 399fb80d407d4bdc87cd1c9c83685b04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:39:07.686124 5902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 399fb80d407d4bdc87cd1c9c83685b04 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:07.687131 5902 raft_consensus.cc:504] T 00000000000000000000000000000000 P 399fb80d407d4bdc87cd1c9c83685b04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399fb80d407d4bdc87cd1c9c83685b04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35169 } }
I20260927 16:39:07.687458 5902 leader_election.cc:302] T 00000000000000000000000000000000 P 399fb80d407d4bdc87cd1c9c83685b04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 399fb80d407d4bdc87cd1c9c83685b04; no voters:
I20260927 16:39:07.687526 5906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:07.678969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50944 (local address 127.0.98.62:35169)
0927 16:39:07.679526 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:07.679541 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:07.679607 (+ 66us) server_negotiation.cc:407] Connection header received
0927 16:39:07.680375 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:07.680396 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:07.680783 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:07.681136 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:07.681529 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:07.682982 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:07.683834 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:07.684588 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:07.684739 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:07.685556 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:07.685563 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:07.685578 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:39:07.686021 (+ 443us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:07.686098 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:39:07.686126 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:07.686225 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:07.686385 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 16:39:07.686628 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":100,"threads_started":1}
I20260927 16:39:07.687784 5902 leader_election.cc:288] T 00000000000000000000000000000000 P 399fb80d407d4bdc87cd1c9c83685b04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:39:07.687868 5907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 399fb80d407d4bdc87cd1c9c83685b04 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:39:07.688062 5907 raft_consensus.cc:686] T 00000000000000000000000000000000 P 399fb80d407d4bdc87cd1c9c83685b04 [term 1 LEADER]: Becoming Leader. State: Replica: 399fb80d407d4bdc87cd1c9c83685b04, State: Running, Role: LEADER
I20260927 16:39:07.688336 5907 consensus_queue.cc:237] T 00000000000000000000000000000000 P 399fb80d407d4bdc87cd1c9c83685b04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399fb80d407d4bdc87cd1c9c83685b04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35169 } }
I20260927 16:39:07.688867 5902 sys_catalog.cc:564] T 00000000000000000000000000000000 P 399fb80d407d4bdc87cd1c9c83685b04 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:39:07.689867 5908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 399fb80d407d4bdc87cd1c9c83685b04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "399fb80d407d4bdc87cd1c9c83685b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399fb80d407d4bdc87cd1c9c83685b04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35169 } } }
I20260927 16:39:07.689975 5908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 399fb80d407d4bdc87cd1c9c83685b04 [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:07.690209 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:39:07.690384 5909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 399fb80d407d4bdc87cd1c9c83685b04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 399fb80d407d4bdc87cd1c9c83685b04. Latest consensus state: current_term: 1 leader_uuid: "399fb80d407d4bdc87cd1c9c83685b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399fb80d407d4bdc87cd1c9c83685b04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35169 } } }
I20260927 16:39:07.690537 5909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 399fb80d407d4bdc87cd1c9c83685b04 [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:07.691777 5916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:39:07.694021 5916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:39:07.698580 5916 catalog_manager.cc:1347] Generated new cluster ID: 4c61b8ee71ce44d8a583f5d8b503482d
I20260927 16:39:07.698642 5916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:39:07.728509 5916 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:39:07.729333 5916 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:39:07.734730 5916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 399fb80d407d4bdc87cd1c9c83685b04: Generated new TSK 0
I20260927 16:39:07.735426 5916 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:39:07.918990 5906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:07.750758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50954 (local address 127.0.98.62:35169)
0927 16:39:07.750896 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:07.750901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:07.750914 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:39:07.750981 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:07.750984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:07.751039 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:07.751140 (+ 101us) server_negotiation.cc:226] Negotiated authn=JWT
0927 16:39:07.751387 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:07.752140 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:07.752914 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:07.753250 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:07.753365 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:07.753614 (+ 249us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 16:39:07.918616 (+165002us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 16:39:07.918861 (+ 245us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50954: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 16:39:07.919056 5906 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50954: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 16:39:07.919281 5931 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:07.750227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35169 (local address 127.0.0.1:50954)
0927 16:39:07.750530 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:07.750547 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:07.750685 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:07.751148 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:07.751150 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:07.751156 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0927 16:39:07.751283 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:07.751289 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:07.752252 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:07.752254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:07.752805 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:07.752811 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:07.753358 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:07.753375 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:07.753432 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:07.753450 (+ 18us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 16:39:07.918833 (+165383us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 16:39:07.919022 (+ 189us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35169: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":94,"threads_started":1}
I20260927 16:39:07.919997 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 5835
2026-09-27T16:39:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (445 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (127 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (111 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:39:08 dist-test-slave-z40b krb5kdc[5943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:39:08 dist-test-slave-z40b krb5kdc[5943](info): set up 2 sockets
Sep 27 16:39:08 dist-test-slave-z40b krb5kdc[5943](info): commencing operation
krb5kdc: starting...
W20260927 16:39:09.159541 392 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:39:09 dist-test-slave-z40b krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:39:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:39:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:39:09.272215 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41135
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:09.373607 5956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:09.373831 5956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:09.373921 5956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:09.377648 5956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:39:09.377717 5956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:09.377751 5956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:09.377808 5956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:39:09.377842 5956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:39:09.382545 5956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:09.383419 5956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:09.384244 5956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:09.392341 5964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:09.392351 5962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:09.392441 5961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:09.392606 5956 server_base.cc:1034] running on GCE node
I20260927 16:39:09.393178 5956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:09.394091 5956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:09.395263 5956 hybrid_clock.cc:648] HybridClock initialized: now 1790527149395231 us; error 43 us; skew 500 ppm
Sep 27 16:39:09 dist-test-slave-z40b krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:09.403404 5956 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:39:09.404196 5956 webserver.cc:460] Webserver started at http://127.0.98.62:39647/ using document root <none> and password file <none>
I20260927 16:39:09.404702 5956 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:09.404786 5956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:09.405001 5956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:09.406850 5956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "d238e460877e4cf3a4177018173bf0ca"
format_stamp: "Formatted at 2026-09-27 16:39:09 on dist-test-slave-z40b"
server_key: "bcfbb330a2da47505e211bd65d80efe9"
server_key_iv: "9951ea3b76b958f70c2271582c966114"
server_key_version: "encryptionkey@0"
I20260927 16:39:09.407451 5956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "d238e460877e4cf3a4177018173bf0ca"
format_stamp: "Formatted at 2026-09-27 16:39:09 on dist-test-slave-z40b"
server_key: "bcfbb330a2da47505e211bd65d80efe9"
server_key_iv: "9951ea3b76b958f70c2271582c966114"
server_key_version: "encryptionkey@0"
I20260927 16:39:09.410722 5956 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 16:39:09.413288 5971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:09.414417 5956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:09.414551 5956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "d238e460877e4cf3a4177018173bf0ca"
format_stamp: "Formatted at 2026-09-27 16:39:09 on dist-test-slave-z40b"
server_key: "bcfbb330a2da47505e211bd65d80efe9"
server_key_iv: "9951ea3b76b958f70c2271582c966114"
server_key_version: "encryptionkey@0"
I20260927 16:39:09.414652 5956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:09.438225 5956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:09.440744 5956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:09.440938 5956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:09.447095 5956 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41135
I20260927 16:39:09.447389 6023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41135 every 8 connection(s)
I20260927 16:39:09.448040 5956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:39:09.450520 6024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:09.455511 6024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d238e460877e4cf3a4177018173bf0ca: Bootstrap starting.
I20260927 16:39:09.457639 6024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d238e460877e4cf3a4177018173bf0ca: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:09.457937 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 5956
I20260927 16:39:09.458060 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:39:09.458319 392 external_mini_cluster.cc:1428] Setting key 96d1991a88f06d7a740b31fc77aac5c3
I20260927 16:39:09.458412 6024 log.cc:824] T 00000000000000000000000000000000 P d238e460877e4cf3a4177018173bf0ca: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:09.460328 6024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d238e460877e4cf3a4177018173bf0ca: No bootstrap required, opened a new log
I20260927 16:39:09.464215 6024 raft_consensus.cc:348] T 00000000000000000000000000000000 P d238e460877e4cf3a4177018173bf0ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d238e460877e4cf3a4177018173bf0ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41135 } }
I20260927 16:39:09.464397 6024 raft_consensus.cc:374] T 00000000000000000000000000000000 P d238e460877e4cf3a4177018173bf0ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:09.464452 6024 raft_consensus.cc:729] T 00000000000000000000000000000000 P d238e460877e4cf3a4177018173bf0ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d238e460877e4cf3a4177018173bf0ca, State: Initialized, Role: FOLLOWER
I20260927 16:39:09.464934 6024 consensus_queue.cc:260] T 00000000000000000000000000000000 P d238e460877e4cf3a4177018173bf0ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d238e460877e4cf3a4177018173bf0ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41135 } }
I20260927 16:39:09.465106 6024 raft_consensus.cc:388] T 00000000000000000000000000000000 P d238e460877e4cf3a4177018173bf0ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:39:09.465188 6024 raft_consensus.cc:482] T 00000000000000000000000000000000 P d238e460877e4cf3a4177018173bf0ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:39:09.465283 6024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d238e460877e4cf3a4177018173bf0ca [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:09.466115 6024 raft_consensus.cc:504] T 00000000000000000000000000000000 P d238e460877e4cf3a4177018173bf0ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d238e460877e4cf3a4177018173bf0ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41135 } }
I20260927 16:39:09.466468 6024 leader_election.cc:302] T 00000000000000000000000000000000 P d238e460877e4cf3a4177018173bf0ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d238e460877e4cf3a4177018173bf0ca; no voters:
Sep 27 16:39:09 dist-test-slave-z40b krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:39:09.466794 6024 leader_election.cc:288] T 00000000000000000000000000000000 P d238e460877e4cf3a4177018173bf0ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:39:09.466935 6029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d238e460877e4cf3a4177018173bf0ca [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:39:09.467171 6029 raft_consensus.cc:686] T 00000000000000000000000000000000 P d238e460877e4cf3a4177018173bf0ca [term 1 LEADER]: Becoming Leader. State: Replica: d238e460877e4cf3a4177018173bf0ca, State: Running, Role: LEADER
I20260927 16:39:09.467509 6029 consensus_queue.cc:237] T 00000000000000000000000000000000 P d238e460877e4cf3a4177018173bf0ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d238e460877e4cf3a4177018173bf0ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41135 } }
I20260927 16:39:09.468093 6024 sys_catalog.cc:564] T 00000000000000000000000000000000 P d238e460877e4cf3a4177018173bf0ca [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:39:09.469050 6031 sys_catalog.cc:455] T 00000000000000000000000000000000 P d238e460877e4cf3a4177018173bf0ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader d238e460877e4cf3a4177018173bf0ca. Latest consensus state: current_term: 1 leader_uuid: "d238e460877e4cf3a4177018173bf0ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d238e460877e4cf3a4177018173bf0ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41135 } } }
I20260927 16:39:09.469163 6031 sys_catalog.cc:458] T 00000000000000000000000000000000 P d238e460877e4cf3a4177018173bf0ca [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:09.469506 6038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:39:09.469486 6030 sys_catalog.cc:455] T 00000000000000000000000000000000 P d238e460877e4cf3a4177018173bf0ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d238e460877e4cf3a4177018173bf0ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d238e460877e4cf3a4177018173bf0ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41135 } } }
I20260927 16:39:09.469586 6030 sys_catalog.cc:458] T 00000000000000000000000000000000 P d238e460877e4cf3a4177018173bf0ca [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:09.472548 6038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:39:09.472757 6028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:09.459374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52738 (local address 127.0.98.62:41135)
0927 16:39:09.459743 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:09.459757 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:09.459788 (+ 31us) server_negotiation.cc:407] Connection header received
0927 16:39:09.460343 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:09.460357 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:09.460685 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:09.460944 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:09.461811 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:09.462686 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:09.463939 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:09.464364 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:09.464531 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:09.467405 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:09.467410 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:09.467424 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:09.467452 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:09.469469 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:09.470038 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:09.470046 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:09.470053 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:09.470201 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:09.470577 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:09.470595 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:09.470598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:09.471073 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:09.471293 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:09.471716 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 16:39:09.471978 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":102,"threads_started":1}
I20260927 16:39:09.477171 6038 catalog_manager.cc:1347] Generated new cluster ID: 634d9a06f8204276aa8649f33a876580
I20260927 16:39:09.477231 6038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:39:09.492514 6038 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:39:09.493269 6038 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:39:09.504241 6038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d238e460877e4cf3a4177018173bf0ca: Generated new TSK 0
I20260927 16:39:09.504848 6038 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:39:09.531576 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41135
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:09.635283 6050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:09.635541 6050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:09.635609 6050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:09.639412 6050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:09.639456 6050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:09.639544 6050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:39:09.644539 6050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:09.645435 6050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:09.646440 6050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:09.655419 6056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:09.655424 6055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:09.655568 6058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:09.655616 6050 server_base.cc:1034] running on GCE node
I20260927 16:39:09.655953 6050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:09.656538 6050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:09.657713 6050 hybrid_clock.cc:648] HybridClock initialized: now 1790527149657702 us; error 32 us; skew 500 ppm
Sep 27 16:39:09 dist-test-slave-z40b krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:09.660265 6050 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:39:09.661046 6050 webserver.cc:460] Webserver started at http://127.0.98.1:37403/ using document root <none> and password file <none>
I20260927 16:39:09.661562 6050 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:09.661646 6050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:09.661881 6050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:09.663857 6050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-0/data/instance:
uuid: "c38905b1eab341f9ba8ec717ce0ade0f"
format_stamp: "Formatted at 2026-09-27 16:39:09 on dist-test-slave-z40b"
server_key: "1901ac4f3cd5fae0111fa1244b0af520"
server_key_iv: "daddc492d857af3b5f173b86bb177ae5"
server_key_version: "encryptionkey@0"
I20260927 16:39:09.664382 6050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c38905b1eab341f9ba8ec717ce0ade0f"
format_stamp: "Formatted at 2026-09-27 16:39:09 on dist-test-slave-z40b"
server_key: "1901ac4f3cd5fae0111fa1244b0af520"
server_key_iv: "daddc492d857af3b5f173b86bb177ae5"
server_key_version: "encryptionkey@0"
I20260927 16:39:09.667709 6050 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:39:09.670042 6065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:09.671139 6050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:39:09.671286 6050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "c38905b1eab341f9ba8ec717ce0ade0f"
format_stamp: "Formatted at 2026-09-27 16:39:09 on dist-test-slave-z40b"
server_key: "1901ac4f3cd5fae0111fa1244b0af520"
server_key_iv: "daddc492d857af3b5f173b86bb177ae5"
server_key_version: "encryptionkey@0"
I20260927 16:39:09.671368 6050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:09.700538 6050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:09.703070 6050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:09.703228 6050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:09.703711 6050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:09.704401 6050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:09.704445 6050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:09.704483 6050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:09.704502 6050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:09.713426 6050 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33471
I20260927 16:39:09.713457 6178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33471 every 8 connection(s)
I20260927 16:39:09.714419 6050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:39:09.716629 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 6050
I20260927 16:39:09.716761 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:39:09.717031 392 external_mini_cluster.cc:1428] Setting key 332b866516ffd0ca3b358b0e6120df0a
Sep 27 16:39:09 dist-test-slave-z40b krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:39:09.727952 6028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:09.716097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45717 (local address 127.0.98.62:41135)
0927 16:39:09.716236 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:09.716241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:09.717086 (+ 845us) server_negotiation.cc:407] Connection header received
0927 16:39:09.718692 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:09.718695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:09.718747 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:09.718821 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:09.720015 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:09.720545 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:09.721652 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:09.721889 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:09.721999 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:09.724487 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:09.724492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:09.724499 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:09.724521 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:09.726004 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:09.726500 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:09.726505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:09.726506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:09.726589 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:09.726909 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:09.726914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:09.726916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:09.727087 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:09.727192 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:09.727669 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 16:39:09.727785 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 16:39:09.728696 6181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:09.716508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41135 (local address 127.0.98.1:45717)
0927 16:39:09.716947 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:09.716995 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:09.718416 (+ 1421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:09.718983 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:09.718989 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:09.719387 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:09.719873 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:09.719888 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:09.720667 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:09.720671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:09.721515 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:09.721524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:09.722030 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:09.722035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:09.722201 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:09.722769 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:09.722784 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:09.724336 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:09.726136 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:09.726142 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:09.726155 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:09.726396 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:09.726681 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:09.726684 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:09.726686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:09.726816 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:09.727189 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:09.727195 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:09.727444 (+ 249us) client_negotiation.cc:769] Sending connection context
0927 16:39:09.727629 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 16:39:09.727914 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":135,"threads_started":1}
I20260927 16:39:09.729858 6179 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41135
I20260927 16:39:09.730094 6179 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:09.730523 6179 heartbeater.cc:511] Master 127.0.98.62:41135 requested a full tablet report, sending...
I20260927 16:39:09.732085 5988 ts_manager.cc:194] Registered new tserver with Master: c38905b1eab341f9ba8ec717ce0ade0f (127.0.98.1:33471)
I20260927 16:39:09.733696 5988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:39:09.742018 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41135
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:09.844230 6184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:09.844444 6184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:09.844481 6184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:09.848331 6184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:09.848373 6184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:09.848423 6184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:39:09.853112 6184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:09.853969 6184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:09.854846 6184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:09.863600 6192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:09.863591 6189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:09.863698 6184 server_base.cc:1034] running on GCE node
W20260927 16:39:09.863590 6190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:09.864153 6184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:09.864704 6184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:09.865872 6184 hybrid_clock.cc:648] HybridClock initialized: now 1790527149865846 us; error 37 us; skew 500 ppm
Sep 27 16:39:09 dist-test-slave-z40b krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:09.868566 6184 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:39:09.869375 6184 webserver.cc:460] Webserver started at http://127.0.98.2:33619/ using document root <none> and password file <none>
I20260927 16:39:09.869851 6184 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:09.869941 6184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:09.870155 6184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:09.872004 6184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-1/data/instance:
uuid: "bc348da024614f2f9d39720aa7f50aaa"
format_stamp: "Formatted at 2026-09-27 16:39:09 on dist-test-slave-z40b"
server_key: "8c0a9ea982fed6ef66c202062e34074d"
server_key_iv: "31a3c62c622b61812e080d72a628f442"
server_key_version: "encryptionkey@0"
I20260927 16:39:09.872428 6184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bc348da024614f2f9d39720aa7f50aaa"
format_stamp: "Formatted at 2026-09-27 16:39:09 on dist-test-slave-z40b"
server_key: "8c0a9ea982fed6ef66c202062e34074d"
server_key_iv: "31a3c62c622b61812e080d72a628f442"
server_key_version: "encryptionkey@0"
I20260927 16:39:09.875617 6184 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:39:09.877616 6199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:09.878662 6184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:09.878765 6184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "bc348da024614f2f9d39720aa7f50aaa"
format_stamp: "Formatted at 2026-09-27 16:39:09 on dist-test-slave-z40b"
server_key: "8c0a9ea982fed6ef66c202062e34074d"
server_key_iv: "31a3c62c622b61812e080d72a628f442"
server_key_version: "encryptionkey@0"
I20260927 16:39:09.878851 6184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:09.897337 6184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:09.900195 6184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:09.900378 6184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:09.900885 6184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:09.901607 6184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:09.901659 6184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:09.901692 6184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:09.901746 6184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:09.910063 6184 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37999
I20260927 16:39:09.910086 6312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37999 every 8 connection(s)
I20260927 16:39:09.910841 6184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:39:09.916993 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 6184
I20260927 16:39:09.917105 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:39:09.917359 392 external_mini_cluster.cc:1428] Setting key a620b483a8d4fcc54ce8282c041e2d67
Sep 27 16:39:09 dist-test-slave-z40b krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:39:09.923604 6028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:09.912411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40241 (local address 127.0.98.62:41135)
0927 16:39:09.912536 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:09.912539 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:09.913238 (+ 699us) server_negotiation.cc:407] Connection header received
0927 16:39:09.914493 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:09.914496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:09.914548 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:09.914641 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:09.915671 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:09.916144 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:09.917186 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:09.917399 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:09.917550 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:09.919951 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:09.919955 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:09.919957 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:09.919977 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:09.921583 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:09.922039 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:09.922045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:09.922050 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:09.922137 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:09.922450 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:09.922455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:09.922459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:09.922683 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:09.922785 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:09.923198 (+ 413us) server_negotiation.cc:299] Negotiation successful
0927 16:39:09.923409 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:39:09.924165 6315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:09.912724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41135 (local address 127.0.98.2:40241)
0927 16:39:09.913103 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:09.913140 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:09.914305 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:09.914791 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:09.914797 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:09.915075 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:09.915551 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:09.915564 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:09.916264 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:09.916267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:09.917058 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:09.917070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:09.917602 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:09.917606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:09.917832 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:09.918334 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:09.918351 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:09.919788 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:09.921699 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:09.921703 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:09.921712 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:09.921940 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:09.922236 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:09.922238 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:09.922239 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:09.922362 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:09.922790 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:09.922794 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:09.922974 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 16:39:09.923105 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 16:39:09.923427 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":100,"threads_started":1}
I20260927 16:39:09.925124 6313 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41135
I20260927 16:39:09.925346 6313 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:09.925786 6313 heartbeater.cc:511] Master 127.0.98.62:41135 requested a full tablet report, sending...
I20260927 16:39:09.926707 5988 ts_manager.cc:194] Registered new tserver with Master: bc348da024614f2f9d39720aa7f50aaa (127.0.98.2:37999)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 16:39:09.927282 5988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:39:09.943845 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41135
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:10.045112 6318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:10.045351 6318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:10.045423 6318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:10.049218 6318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:10.049293 6318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:10.049378 6318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:39:10.054272 6318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:10.055138 6318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:10.056044 6318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:10.064479 6323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:10.064457 6326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:10.064549 6318 server_base.cc:1034] running on GCE node
W20260927 16:39:10.064476 6324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:10.065014 6318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:10.065618 6318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:10.066794 6318 hybrid_clock.cc:648] HybridClock initialized: now 1790527150066766 us; error 41 us; skew 500 ppm
Sep 27 16:39:10 dist-test-slave-z40b krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:10.069478 6318 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:39:10.070238 6318 webserver.cc:460] Webserver started at http://127.0.98.3:38159/ using document root <none> and password file <none>
I20260927 16:39:10.070739 6318 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:10.070822 6318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:10.071041 6318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:10.072898 6318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-2/data/instance:
uuid: "b5086aa9547541fba72df4d0a19bddbf"
format_stamp: "Formatted at 2026-09-27 16:39:10 on dist-test-slave-z40b"
server_key: "b434b6551a41b6996c0d0c44a2d5cb65"
server_key_iv: "888ac78a107733f89023b2cfa8813f8d"
server_key_version: "encryptionkey@0"
I20260927 16:39:10.073421 6318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b5086aa9547541fba72df4d0a19bddbf"
format_stamp: "Formatted at 2026-09-27 16:39:10 on dist-test-slave-z40b"
server_key: "b434b6551a41b6996c0d0c44a2d5cb65"
server_key_iv: "888ac78a107733f89023b2cfa8813f8d"
server_key_version: "encryptionkey@0"
I20260927 16:39:10.076812 6318 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 16:39:10.079031 6333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:10.080191 6318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:39:10.080305 6318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "b5086aa9547541fba72df4d0a19bddbf"
format_stamp: "Formatted at 2026-09-27 16:39:10 on dist-test-slave-z40b"
server_key: "b434b6551a41b6996c0d0c44a2d5cb65"
server_key_iv: "888ac78a107733f89023b2cfa8813f8d"
server_key_version: "encryptionkey@0"
I20260927 16:39:10.080399 6318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:10.107161 6318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:10.109845 6318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:10.110035 6318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:10.110569 6318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:10.111363 6318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:10.111405 6318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:10.111469 6318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:10.111509 6318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:10.120020 6318 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40507
I20260927 16:39:10.120048 6446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40507 every 8 connection(s)
I20260927 16:39:10.120810 6318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:39:10.129220 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 6318
I20260927 16:39:10.129350 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/minicluster-data/ts-2/wal/instance
Sep 27 16:39:10 dist-test-slave-z40b krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:39:10.129642 392 external_mini_cluster.cc:1428] Setting key 9e1e9c7f306b9cb34627266e88ffe14f
I20260927 16:39:10.133569 6028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:10.122391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59605 (local address 127.0.98.62:41135)
0927 16:39:10.122518 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:10.122522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:10.123293 (+ 771us) server_negotiation.cc:407] Connection header received
0927 16:39:10.124603 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:10.124606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:10.124669 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:10.124769 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:10.125859 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:10.126384 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:10.127545 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:10.127763 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:10.127870 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:10.130186 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:10.130190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:10.130192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:10.130220 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:10.131678 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:10.132188 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:10.132192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:10.132194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:10.132268 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:10.132578 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:10.132582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:10.132584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:10.132804 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:10.132917 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:10.133328 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 16:39:10.133425 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:39:10.134217 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:10.122734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41135 (local address 127.0.98.3:59605)
0927 16:39:10.123121 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:10.123164 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:10.124400 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:10.124947 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:10.124954 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:10.125273 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:10.125710 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:10.125722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:10.126503 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:10.126506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:10.127418 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:10.127426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:10.127901 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:10.127905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:10.128081 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:10.128553 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:10.128567 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:10.130038 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:10.131824 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:10.131829 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:10.131841 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:10.132072 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:10.132368 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:10.132371 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:10.132373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:10.132492 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:10.132917 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:10.132923 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:10.133125 (+ 202us) client_negotiation.cc:769] Sending connection context
0927 16:39:10.133291 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 16:39:10.133548 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":116,"threads_started":1}
I20260927 16:39:10.135131 6447 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41135
I20260927 16:39:10.135368 6447 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:10.135794 6447 heartbeater.cc:511] Master 127.0.98.62:41135 requested a full tablet report, sending...
I20260927 16:39:10.136696 5988 ts_manager.cc:194] Registered new tserver with Master: b5086aa9547541fba72df4d0a19bddbf (127.0.98.3:40507)
I20260927 16:39:10.137183 5988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59605
I20260927 16:39:10.143077 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/test-admin.keytab.
Sep 27 16:39:10 dist-test-slave-z40b krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:10.157606 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 16:39:10.160074 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/krb5cc' of type FILE
W20260927 16:39:10.160094 6454 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:10.158796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58826 (local address 127.0.98.1:33471)
0927 16:39:10.159071 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:10.159079 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:10.159099 (+ 20us) server_negotiation.cc:407] Connection header received
0927 16:39:10.159141 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:10.159145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:10.159358 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:10.159466 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:10.159914 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58826: BlockingRecv error: recv got EOF from 127.0.0.1:58826 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":99,"threads_started":1}
Sep 27 16:39:10 dist-test-slave-z40b krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:10.161008 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 16:39:10 dist-test-slave-z40b krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:10.161808 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 16:39:10 dist-test-slave-z40b krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 16:39:10.170431 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:10.162493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58834 (local address 127.0.98.1:33471)
0927 16:39:10.162594 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:10.162597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:10.162766 (+ 169us) server_negotiation.cc:407] Connection header received
0927 16:39:10.162825 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:10.162827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:10.162860 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:10.162916 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:10.163415 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:10.164152 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:10.164907 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:10.165237 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:10.165315 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:10.167402 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:10.167410 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:10.167419 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:10.167437 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:10.168864 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:10.169219 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:10.169223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:10.169224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:10.169289 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:10.169563 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:10.169566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:10.169568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:10.169828 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:10.169969 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:10.170118 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 16:39:10.170272 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 16:39:10.173612 6454 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:10.172903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58846 (local address 127.0.98.1:33471)
0927 16:39:10.173015 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:10.173018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:10.173029 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:39:10.173115 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:10.173117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:10.173144 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:10.173206 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:10.173528 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58846: BlockingRecv error: recv got EOF from 127.0.0.1:58846 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 16:39:10.173702 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 16:39:10 dist-test-slave-z40b krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:10.174584 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 16:39:10 dist-test-slave-z40b krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:10.175458 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 16:39:10 dist-test-slave-z40b krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 16:39:10.182832 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:10.176145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58852 (local address 127.0.98.1:33471)
0927 16:39:10.176242 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:10.176245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:10.176351 (+ 106us) server_negotiation.cc:407] Connection header received
0927 16:39:10.176411 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:10.176413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:10.176438 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:10.176507 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:10.176900 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:10.177360 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:10.178032 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:10.178279 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:10.178351 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:10.180385 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:10.180387 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:10.180389 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:10.180408 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:10.181592 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:10.181931 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:10.181933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:10.181934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:10.182000 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:10.182224 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:10.182226 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:10.182227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:10.182349 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:10.182489 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:10.182668 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 16:39:10.182745 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 16:39:10.185099 6454 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:10.184231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58858 (local address 127.0.98.1:33471)
0927 16:39:10.184330 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:10.184334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:10.184463 (+ 129us) server_negotiation.cc:407] Connection header received
0927 16:39:10.184583 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:10.184585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:10.184630 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:10.184700 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:10.184987 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58858: BlockingRecv error: recv got EOF from 127.0.0.1:58858 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 16:39:10.185240 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790527121515808-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 16:39:10 dist-test-slave-z40b krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:10.186123 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 16:39:10 dist-test-slave-z40b krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:10.186964 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 16:39:10 dist-test-slave-z40b krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 16:39:10.194263 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:10.187710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58874 (local address 127.0.98.1:33471)
0927 16:39:10.187809 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:10.187812 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:10.187939 (+ 127us) server_negotiation.cc:407] Connection header received
0927 16:39:10.188015 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:10.188017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:10.188046 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:10.188108 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:10.188545 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:10.188984 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:10.189693 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:10.189978 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:10.190053 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:10.192052 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:10.192055 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:10.192057 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:10.192074 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:10.193137 (+ 1063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:10.193467 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:10.193469 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:10.193470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:10.193530 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:10.193733 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:10.193735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:10.193736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:10.193854 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:10.193948 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:10.194103 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 16:39:10.194175 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 16:39:10.195188 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 6050
I20260927 16:39:10.200938 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 6184
I20260927 16:39:10.206071 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 6318
I20260927 16:39:10.212574 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 5956
2026-09-27T16:39:10Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2054 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:39:10 dist-test-slave-z40b krb5kdc[6467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:39:10 dist-test-slave-z40b krb5kdc[6467](info): set up 2 sockets
Sep 27 16:39:10 dist-test-slave-z40b krb5kdc[6467](info): commencing operation
krb5kdc: starting...
W20260927 16:39:11.219347 392 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:39:11 dist-test-slave-z40b krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:39:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:39:11Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 16:39:11.330085 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41911
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41911
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:11.430778 6480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:11.430994 6480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:11.431032 6480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:11.434805 6480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:39:11.434852 6480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:11.434871 6480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:11.434930 6480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:39:11.434958 6480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:39:11.440150 6480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:11.441049 6480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:11.441975 6480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:11.449815 6488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:11.449812 6485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:11.449923 6486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:11.450222 6480 server_base.cc:1034] running on GCE node
I20260927 16:39:11.450765 6480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:11.451853 6480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:11.453028 6480 hybrid_clock.cc:648] HybridClock initialized: now 1790527151453023 us; error 28 us; skew 500 ppm
Sep 27 16:39:11 dist-test-slave-z40b krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527151, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:11.455775 6480 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 16:39:11.456534 6480 webserver.cc:460] Webserver started at http://127.0.98.62:43303/ using document root <none> and password file <none>
I20260927 16:39:11.457062 6480 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:11.457118 6480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:11.457330 6480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:11.459304 6480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "d0dece26d2544ea59c76a1edb6ee1ade"
format_stamp: "Formatted at 2026-09-27 16:39:11 on dist-test-slave-z40b"
server_key: "6087a3b898436cd2d1cd0bb9eccd7e54"
server_key_iv: "37ad66755d3ad753be1441e59bb74f45"
server_key_version: "encryptionkey@0"
I20260927 16:39:11.459860 6480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "d0dece26d2544ea59c76a1edb6ee1ade"
format_stamp: "Formatted at 2026-09-27 16:39:11 on dist-test-slave-z40b"
server_key: "6087a3b898436cd2d1cd0bb9eccd7e54"
server_key_iv: "37ad66755d3ad753be1441e59bb74f45"
server_key_version: "encryptionkey@0"
I20260927 16:39:11.463284 6480 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 16:39:11.465428 6495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:11.466512 6480 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:11.466629 6480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "d0dece26d2544ea59c76a1edb6ee1ade"
format_stamp: "Formatted at 2026-09-27 16:39:11 on dist-test-slave-z40b"
server_key: "6087a3b898436cd2d1cd0bb9eccd7e54"
server_key_iv: "37ad66755d3ad753be1441e59bb74f45"
server_key_version: "encryptionkey@0"
I20260927 16:39:11.466729 6480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:11.493417 6480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:11.496480 6480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:11.496657 6480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:11.502880 6480 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41911
I20260927 16:39:11.503156 6547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41911 every 8 connection(s)
I20260927 16:39:11.503773 6480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:39:11.505687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 6480
I20260927 16:39:11.505833 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:39:11.506110 392 external_mini_cluster.cc:1428] Setting key 4aad8992b26946f8fbe72193c6e7547e
I20260927 16:39:11.506402 6548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:11.512079 6548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade: Bootstrap starting.
Sep 27 16:39:11 dist-test-slave-z40b krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 16:39:11.514401 6548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:11.515060 6548 log.cc:824] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:11.517313 6548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade: No bootstrap required, opened a new log
I20260927 16:39:11.519776 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:11.507374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33074 (local address 127.0.98.62:41911)
0927 16:39:11.507767 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:11.507781 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:11.507832 (+ 51us) server_negotiation.cc:407] Connection header received
0927 16:39:11.508440 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:11.508455 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:11.508744 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:11.509064 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:11.509801 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:11.510751 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:11.511605 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:11.512066 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:11.512258 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:11.514766 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:11.514771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:11.514782 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:11.514805 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:11.516825 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:11.517219 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:11.517225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:11.517230 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:11.517364 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:11.517667 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:11.517670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:11.517672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:11.518156 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:11.518342 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:11.518525 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 16:39:11.518790 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":125,"threads_started":1}
I20260927 16:39:11.521522 6548 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } }
I20260927 16:39:11.521791 6548 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:11.521852 6548 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dece26d2544ea59c76a1edb6ee1ade, State: Initialized, Role: FOLLOWER
I20260927 16:39:11.522264 6548 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } }
I20260927 16:39:11.522405 6548 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:39:11.522487 6548 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:39:11.522588 6548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:11.523541 6548 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } }
I20260927 16:39:11.523857 6548 leader_election.cc:302] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0dece26d2544ea59c76a1edb6ee1ade; no voters:
I20260927 16:39:11.524116 6548 leader_election.cc:288] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:39:11.524266 6553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:39:11.524477 6553 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [term 1 LEADER]: Becoming Leader. State: Replica: d0dece26d2544ea59c76a1edb6ee1ade, State: Running, Role: LEADER
I20260927 16:39:11.524808 6553 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } }
I20260927 16:39:11.525225 6548 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:39:11.526248 6554 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } }
I20260927 16:39:11.526363 6554 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:11.526719 6555 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0dece26d2544ea59c76a1edb6ee1ade. Latest consensus state: current_term: 1 leader_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } }
I20260927 16:39:11.526872 6555 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:11.527496 6562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:39:11.529841 6562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:39:11.534814 6562 catalog_manager.cc:1347] Generated new cluster ID: 7b929cdf13ac47ff88f0adc0945d6ec6
I20260927 16:39:11.534878 6562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:39:11.543529 6562 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:39:11.544255 6562 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:39:11.550531 6562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade: Generated new TSK 0
I20260927 16:39:11.551185 6562 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 16:39:11.582157 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41911
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:11.686206 6574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:11.686429 6574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:11.686496 6574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:11.690266 6574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:11.690316 6574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:11.690402 6574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:39:11.695350 6574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:11.696220 6574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:11.697129 6574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:11.706136 6580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:11.706148 6579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:11.706197 6574 server_base.cc:1034] running on GCE node
W20260927 16:39:11.706148 6582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:11.706722 6574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:11.707351 6574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:11.708526 6574 hybrid_clock.cc:648] HybridClock initialized: now 1790527151708506 us; error 37 us; skew 500 ppm
Sep 27 16:39:11 dist-test-slave-z40b krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527151, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:11.711102 6574 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 16:39:11.711931 6574 webserver.cc:460] Webserver started at http://127.0.98.1:41617/ using document root <none> and password file <none>
I20260927 16:39:11.712443 6574 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:11.712530 6574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:11.712755 6574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:11.714659 6574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/data/instance:
uuid: "c7c8dd9decf94b0eb2498082d63112a3"
format_stamp: "Formatted at 2026-09-27 16:39:11 on dist-test-slave-z40b"
server_key: "86436f94d3260fffc8250d4be93d397b"
server_key_iv: "eafc2c41dd2056ec5a07ac946e49d54e"
server_key_version: "encryptionkey@0"
I20260927 16:39:11.715276 6574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c7c8dd9decf94b0eb2498082d63112a3"
format_stamp: "Formatted at 2026-09-27 16:39:11 on dist-test-slave-z40b"
server_key: "86436f94d3260fffc8250d4be93d397b"
server_key_iv: "eafc2c41dd2056ec5a07ac946e49d54e"
server_key_version: "encryptionkey@0"
I20260927 16:39:11.718556 6574 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:39:11.720769 6589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:11.721835 6574 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:39:11.721958 6574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "c7c8dd9decf94b0eb2498082d63112a3"
format_stamp: "Formatted at 2026-09-27 16:39:11 on dist-test-slave-z40b"
server_key: "86436f94d3260fffc8250d4be93d397b"
server_key_iv: "eafc2c41dd2056ec5a07ac946e49d54e"
server_key_version: "encryptionkey@0"
I20260927 16:39:11.722056 6574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:11.737556 6574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:11.740449 6574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:11.740633 6574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:11.741146 6574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:11.741865 6574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:11.741930 6574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:11.741989 6574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:11.742039 6574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:11.750115 6574 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42395
I20260927 16:39:11.750181 6702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42395 every 8 connection(s)
I20260927 16:39:11.750926 6574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:39:11.757334 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 6574
I20260927 16:39:11.757474 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:39:11.757750 392 external_mini_cluster.cc:1428] Setting key ac6945bef90c25d5e20f2761c3171351
Sep 27 16:39:11 dist-test-slave-z40b krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527151, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 16:39:11.764413 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:11.752544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44719 (local address 127.0.98.62:41911)
0927 16:39:11.752663 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:11.752668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:11.753446 (+ 778us) server_negotiation.cc:407] Connection header received
0927 16:39:11.754701 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:11.754704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:11.754766 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:11.754861 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:11.755992 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:11.756522 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:11.757667 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:11.757978 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:11.758085 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:11.760671 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:11.760674 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:11.760676 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:11.760698 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:11.762303 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:11.762775 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:11.762781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:11.762786 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:11.762876 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:11.763178 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:11.763184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:11.763189 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:11.763461 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:11.763579 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:11.764025 (+ 446us) server_negotiation.cc:299] Negotiation successful
0927 16:39:11.764169 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:39:11.764947 6705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:11.752904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.1:44719)
0927 16:39:11.753293 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:11.753330 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:11.754490 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:11.755015 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:11.755021 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:11.755411 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:11.755846 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:11.755858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:11.756641 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:11.756645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:11.757531 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:11.757556 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:11.758117 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:11.758121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:11.758285 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:11.758764 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:11.758778 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:11.760523 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:11.762413 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:11.762418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:11.762428 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:11.762677 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:11.762967 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:11.762969 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:11.762971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:11.763085 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:11.763590 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:11.763595 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:11.763788 (+ 193us) client_negotiation.cc:769] Sending connection context
0927 16:39:11.763946 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 16:39:11.764158 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":93,"threads_started":1}
I20260927 16:39:11.766103 6703 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41911
I20260927 16:39:11.766335 6703 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:11.766871 6703 heartbeater.cc:511] Master 127.0.98.62:41911 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:39:11.768370 6512 ts_manager.cc:194] Registered new tserver with Master: c7c8dd9decf94b0eb2498082d63112a3 (127.0.98.1:42395)
I20260927 16:39:11.769630 6512 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 16:39:11.785816 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41911
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:11.888186 6708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:11.888420 6708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:11.888458 6708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:11.892112 6708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:11.892150 6708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:11.892197 6708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:39:11.896708 6708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:11.897534 6708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:11.898370 6708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:11.907346 6716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:11.907351 6714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:11.907440 6708 server_base.cc:1034] running on GCE node
W20260927 16:39:11.907327 6713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:11.908000 6708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:11.908613 6708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:11.909785 6708 hybrid_clock.cc:648] HybridClock initialized: now 1790527151909761 us; error 43 us; skew 500 ppm
Sep 27 16:39:11 dist-test-slave-z40b krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527151, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:11.912345 6708 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 16:39:11.913172 6708 webserver.cc:460] Webserver started at http://127.0.98.2:45501/ using document root <none> and password file <none>
I20260927 16:39:11.913688 6708 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:11.913772 6708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:11.913975 6708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:11.915843 6708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/data/instance:
uuid: "aa1787383203493a93865001e604596f"
format_stamp: "Formatted at 2026-09-27 16:39:11 on dist-test-slave-z40b"
server_key: "ce54e4a2d2da0bf1b8b3d55689e8d60f"
server_key_iv: "4ace39de14582b6a1b3f0de2514aa02f"
server_key_version: "encryptionkey@0"
I20260927 16:39:11.916388 6708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/wal/instance:
uuid: "aa1787383203493a93865001e604596f"
format_stamp: "Formatted at 2026-09-27 16:39:11 on dist-test-slave-z40b"
server_key: "ce54e4a2d2da0bf1b8b3d55689e8d60f"
server_key_iv: "4ace39de14582b6a1b3f0de2514aa02f"
server_key_version: "encryptionkey@0"
I20260927 16:39:11.919834 6708 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 16:39:11.922003 6723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:11.923138 6708 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:39:11.923295 6708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "aa1787383203493a93865001e604596f"
format_stamp: "Formatted at 2026-09-27 16:39:11 on dist-test-slave-z40b"
server_key: "ce54e4a2d2da0bf1b8b3d55689e8d60f"
server_key_iv: "4ace39de14582b6a1b3f0de2514aa02f"
server_key_version: "encryptionkey@0"
I20260927 16:39:11.923396 6708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:11.936913 6708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:11.939455 6708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:11.939632 6708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:11.940155 6708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:11.940871 6708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:11.940939 6708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:11.941006 6708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:11.941057 6708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:11.949437 6708 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45479
I20260927 16:39:11.949460 6836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45479 every 8 connection(s)
I20260927 16:39:11.950201 6708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:39:11.951071 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 6708
I20260927 16:39:11.951195 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:39:11.951491 392 external_mini_cluster.cc:1428] Setting key e47ece88f8f021db9299ff7ca3c2fc25
Sep 27 16:39:11 dist-test-slave-z40b krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527151, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260927 16:39:11.964802 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:11.952153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39637 (local address 127.0.98.62:41911)
0927 16:39:11.952294 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:11.952300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:11.953009 (+ 709us) server_negotiation.cc:407] Connection header received
0927 16:39:11.954903 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:11.954908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:11.954966 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:11.955051 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:11.956051 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:11.956603 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:11.958367 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:11.958566 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:11.958640 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:11.961300 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:11.961304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:11.961306 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:11.961332 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:11.962786 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:11.963278 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:11.963281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:11.963283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:11.963363 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:11.963752 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:11.963754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:11.963755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:11.963910 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:11.963983 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:11.964552 (+ 569us) server_negotiation.cc:299] Negotiation successful
0927 16:39:11.964666 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 16:39:11.965979 6839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:11.952328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.2:39637)
0927 16:39:11.952834 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:11.952889 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:11.954660 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:11.955176 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:11.955182 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:11.955515 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:11.955924 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:11.955935 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:11.956751 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:11.956757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:11.958246 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:11.958260 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:11.958743 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:11.958751 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:11.958979 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:11.959578 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:11.959602 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:11.961142 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:11.962880 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:11.962884 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:11.962894 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:11.963143 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:11.963479 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:11.963485 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:11.963490 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:11.963666 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:11.963994 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:11.963998 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:11.964322 (+ 324us) client_negotiation.cc:769] Sending connection context
0927 16:39:11.964467 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 16:39:11.964833 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":128,"threads_started":1}
I20260927 16:39:11.967034 6837 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41911
I20260927 16:39:11.967281 6837 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:11.967770 6837 heartbeater.cc:511] Master 127.0.98.62:41911 requested a full tablet report, sending...
I20260927 16:39:11.968889 6512 ts_manager.cc:194] Registered new tserver with Master: aa1787383203493a93865001e604596f (127.0.98.2:45479)
I20260927 16:39:11.969745 6512 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39637
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 16:39:12.004492 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41911
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:12.107348 6842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:12.107563 6842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:12.107602 6842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:12.111497 6842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:12.111545 6842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:12.111617 6842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:39:12.116576 6842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:12.117424 6842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:12.118291 6842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:12.127246 6847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:12.127246 6850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:12.127424 6842 server_base.cc:1034] running on GCE node
W20260927 16:39:12.127472 6848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:12.127815 6842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:12.128403 6842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:12.129578 6842 hybrid_clock.cc:648] HybridClock initialized: now 1790527152129566 us; error 33 us; skew 500 ppm
Sep 27 16:39:12 dist-test-slave-z40b krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527152, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:12.132081 6842 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 16:39:12.132826 6842 webserver.cc:460] Webserver started at http://127.0.98.3:39631/ using document root <none> and password file <none>
I20260927 16:39:12.133324 6842 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:12.133409 6842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:12.133625 6842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:12.135612 6842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/data/instance:
uuid: "b1f3d8ccb5e64924a7327a038fed3756"
format_stamp: "Formatted at 2026-09-27 16:39:12 on dist-test-slave-z40b"
server_key: "f2a44b65eda2d9259ccbd7db172cd8d4"
server_key_iv: "9e951118032293981070ecd91b7d76ae"
server_key_version: "encryptionkey@0"
I20260927 16:39:12.136139 6842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b1f3d8ccb5e64924a7327a038fed3756"
format_stamp: "Formatted at 2026-09-27 16:39:12 on dist-test-slave-z40b"
server_key: "f2a44b65eda2d9259ccbd7db172cd8d4"
server_key_iv: "9e951118032293981070ecd91b7d76ae"
server_key_version: "encryptionkey@0"
I20260927 16:39:12.139747 6842 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:39:12.141909 6857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:12.143006 6842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:12.143129 6842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "b1f3d8ccb5e64924a7327a038fed3756"
format_stamp: "Formatted at 2026-09-27 16:39:12 on dist-test-slave-z40b"
server_key: "f2a44b65eda2d9259ccbd7db172cd8d4"
server_key_iv: "9e951118032293981070ecd91b7d76ae"
server_key_version: "encryptionkey@0"
I20260927 16:39:12.143247 6842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:12.152225 6842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:12.154572 6842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:12.154747 6842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:12.155355 6842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:12.156068 6842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:12.156132 6842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:12.156193 6842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:12.156234 6842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:12.164335 6842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33697
I20260927 16:39:12.164357 6970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33697 every 8 connection(s)
I20260927 16:39:12.165153 6842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:39:12.169919 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 6842
I20260927 16:39:12.170018 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:39:12.170347 392 external_mini_cluster.cc:1428] Setting key d88e614fc788f30fb6e1fdf13d06f2fe
Sep 27 16:39:12 dist-test-slave-z40b krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527152, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 16:39:12.177790 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.166767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57223 (local address 127.0.98.62:41911)
0927 16:39:12.166891 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:12.166895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:12.167663 (+ 768us) server_negotiation.cc:407] Connection header received
0927 16:39:12.168981 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:12.168984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:12.169042 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:12.169118 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:12.170479 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.171266 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.172601 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.172808 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.172894 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.175031 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:12.175034 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:12.175036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:12.175054 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:12.176281 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.176699 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.176701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.176702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.176779 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.177038 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.177040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.177041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.177181 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:12.177246 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:12.177597 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 16:39:12.177675 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:39:12.178424 6973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.167091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.3:57223)
0927 16:39:12.167536 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:12.167572 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:12.168748 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:12.169285 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:12.169291 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:12.169703 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:12.170325 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.170342 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.171395 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.171401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.172506 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.172514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.172928 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.172930 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.173044 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.173529 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:12.173542 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:12.174907 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:12.176370 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.176374 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.176383 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.176621 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.176851 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.176853 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.176854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.176971 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.177255 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:12.177259 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:12.177430 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 16:39:12.177559 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 16:39:12.177764 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":161,"threads_started":1}
I20260927 16:39:12.179380 6971 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41911
I20260927 16:39:12.179584 6971 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:12.180017 6971 heartbeater.cc:511] Master 127.0.98.62:41911 requested a full tablet report, sending...
I20260927 16:39:12.180895 6512 ts_manager.cc:194] Registered new tserver with Master: b1f3d8ccb5e64924a7327a038fed3756 (127.0.98.3:33697)
I20260927 16:39:12.181398 6512 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57223
I20260927 16:39:12.183840 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260927 16:39:12.207981 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46569
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41911,127.0.98.61:46569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:12.311447 6976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:12.311663 6976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:12.311702 6976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:12.315541 6976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:39:12.315587 6976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:12.315606 6976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:12.315622 6976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:39:12.315636 6976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:39:12.320567 6976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41911,127.0.98.61:46569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
W20260927 16:39:12.320722 6976 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:41911,127.0.98.61:46569'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 16:39:12.321429 6976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:12.322253 6976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:12.330523 6982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:12.330578 6976 server_base.cc:1034] running on GCE node
W20260927 16:39:12.330521 6981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:12.330529 6984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:12.331310 6976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:12.332298 6976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:12.333483 6976 hybrid_clock.cc:648] HybridClock initialized: now 1790527152333463 us; error 28 us; skew 500 ppm
Sep 27 16:39:12 dist-test-slave-z40b krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527152, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:12.335963 6976 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 16:39:12.336740 6976 webserver.cc:460] Webserver started at http://127.0.98.61:39355/ using document root <none> and password file <none>
I20260927 16:39:12.337244 6976 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:12.337327 6976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:12.337618 6976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:12.339596 6976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-1/data/instance:
uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed"
format_stamp: "Formatted at 2026-09-27 16:39:12 on dist-test-slave-z40b"
server_key: "8ad427dd6034fe18a07e7d01fa0a3944"
server_key_iv: "6ee2fa703a989734beac7d9a34d0e07a"
server_key_version: "encryptionkey@0"
I20260927 16:39:12.340142 6976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-1/wal/instance:
uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed"
format_stamp: "Formatted at 2026-09-27 16:39:12 on dist-test-slave-z40b"
server_key: "8ad427dd6034fe18a07e7d01fa0a3944"
server_key_iv: "6ee2fa703a989734beac7d9a34d0e07a"
server_key_version: "encryptionkey@0"
I20260927 16:39:12.343436 6976 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 16:39:12.345541 6991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:12.346719 6976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:12.346846 6976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-1/wal
uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed"
format_stamp: "Formatted at 2026-09-27 16:39:12 on dist-test-slave-z40b"
server_key: "8ad427dd6034fe18a07e7d01fa0a3944"
server_key_iv: "6ee2fa703a989734beac7d9a34d0e07a"
server_key_version: "encryptionkey@0"
I20260927 16:39:12.346946 6976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:12.357733 6976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:12.360365 6976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:12.360546 6976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:12.367009 6976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46569
I20260927 16:39:12.367336 7043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46569 every 8 connection(s)
I20260927 16:39:12.367990 6976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-1/data/info.pb
I20260927 16:39:12.370661 7044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:12.372970 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 6976
I20260927 16:39:12.374581 7044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:39:12 dist-test-slave-z40b krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527152, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 16:39:12 dist-test-slave-z40b krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 16:39:12.386746 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.376226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33082 (local address 127.0.98.62:41911)
0927 16:39:12.376373 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:12.376379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:12.376801 (+ 422us) server_negotiation.cc:407] Connection header received
0927 16:39:12.376913 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:12.376915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:12.376958 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:12.377021 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:12.377706 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.378471 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.379784 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.380126 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.380216 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.382576 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:12.382581 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:12.382583 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:12.382616 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:12.384308 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.384910 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.384913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.384915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.385015 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.385342 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.385344 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.385345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.385700 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:12.385787 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:12.386314 (+ 527us) server_negotiation.cc:299] Negotiation successful
0927 16:39:12.386437 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 16:39:12.387461 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.374376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58566 (local address 127.0.98.61:46569)
0927 16:39:12.374765 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:12.374782 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:12.374815 (+ 33us) server_negotiation.cc:407] Connection header received
0927 16:39:12.375525 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:12.375542 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:12.375878 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:12.376189 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:12.377128 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.378392 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.379545 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.380036 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.380201 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.383095 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:12.383101 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:12.383120 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:12.383139 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:12.384808 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.385257 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.385261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.385263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.385338 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.385663 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.385665 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.385666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.386096 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:12.386286 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:12.386491 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 16:39:12.386597 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":11,"server-negotiator.queue_time_us":237,"spinlock_wait_cycles":3712,"thread_start_us":116,"threads_started":1}
I20260927 16:39:12.387461 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.376144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.0.1:33082)
0927 16:39:12.376638 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:12.376689 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:12.376822 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:12.377126 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:12.377130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:12.377424 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:12.377562 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.377570 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.378607 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.378611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.379668 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.379676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.380270 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.380274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.380379 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.380861 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:12.380885 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:12.382417 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:12.384409 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.384415 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.384423 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.384808 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.385101 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.385103 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.385104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.385251 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.385805 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:12.385810 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:12.386096 (+ 286us) client_negotiation.cc:769] Sending connection context
0927 16:39:12.386257 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 16:39:12.386553 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":37,"spinlock_wait_cycles":17152,"thread_start_us":88,"threads_started":1}
I20260927 16:39:12.388798 7044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:39:12 dist-test-slave-z40b krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527152, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 16:39:12.395902 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.389252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46569 (local address 127.0.0.1:58574)
0927 16:39:12.389340 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:12.389353 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:12.389414 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:12.389625 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:12.389628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:12.389776 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:12.389842 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.389847 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.390531 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.390537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.391021 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.391027 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.391932 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.391935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.392022 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.392431 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:12.392443 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:12.393511 (+ 1068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:12.394802 (+ 1291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.394804 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.394805 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.394975 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.395192 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.395194 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.395195 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.395305 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.395588 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:12.395590 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:12.395652 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 16:39:12.395732 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 16:39:12.395810 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 16:39:12.396062 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.389289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58574 (local address 127.0.98.61:46569)
0927 16:39:12.389403 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:12.389409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:12.389423 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:39:12.389482 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:12.389484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:12.389525 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:12.389616 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:12.389945 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.390433 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.391128 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.391815 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.391935 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.393621 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:12.393624 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:12.393625 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:12.393644 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:12.394712 (+ 1068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.395052 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.395054 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.395055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.395116 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.395378 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.395380 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.395380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.395513 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:12.395583 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:12.395785 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 16:39:12.395874 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":39}
I20260927 16:39:12.398211 7044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed: Bootstrap starting.
I20260927 16:39:12.400206 7044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:12.400835 7044 log.cc:824] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:12.402405 7044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed: No bootstrap required, opened a new log
I20260927 16:39:12.405119 7044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } }
I20260927 16:39:12.405282 7044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:12.405320 7044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed, State: Initialized, Role: FOLLOWER
I20260927 16:39:12.405709 7044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } }
I20260927 16:39:12.406071 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } } }
I20260927 16:39:12.406173 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:39:12.406692 7044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [sys.catalog]: configured and running, proceeding with master startup.
W20260927 16:39:12.410781 7061 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:39:12.410868 7061 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:39:12.412448 6976 master_runner.cc:428] Detected that this master 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed is joining an existing cluster
I20260927 16:39:12.412509 6976 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:46569
I20260927 16:39:12.422497 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.414321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33088 (local address 127.0.98.62:41911)
0927 16:39:12.414440 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:12.414445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:12.414674 (+ 229us) server_negotiation.cc:407] Connection header received
0927 16:39:12.414741 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:12.414745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:12.414789 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:12.414861 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:12.415377 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.416181 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.417327 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.417668 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.417782 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.418964 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:12.418968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:12.418971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:12.418996 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:12.420753 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.421269 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.421272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.421274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.421358 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.421666 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.421672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.421676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.421889 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:12.421960 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:12.422252 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 16:39:12.422353 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 16:39:12.422502 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.414263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.0.1:33088)
0927 16:39:12.414531 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:12.414543 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:12.414655 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:12.414872 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:12.414874 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:12.415086 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:12.415183 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.415188 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.416311 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.416314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.417203 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.417210 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.417788 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.417791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.417877 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.418380 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:12.418393 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:12.418829 (+ 436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:12.420866 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.420870 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.420872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.421154 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.421460 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.421465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.421470 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.421563 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.422024 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:12.422026 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:12.422101 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 16:39:12.422201 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 16:39:12.422316 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"spinlock_wait_cycles":6528,"thread_start_us":102,"threads_started":1}
I20260927 16:39:12.422956 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.414704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58580 (local address 127.0.98.61:46569)
0927 16:39:12.414800 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:12.414804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:12.415470 (+ 666us) server_negotiation.cc:407] Connection header received
0927 16:39:12.415594 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:12.415596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:12.415627 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:12.415667 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:12.416176 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.416915 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.417767 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.418097 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.418151 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.419761 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:12.419765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:12.419767 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:12.419786 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:12.421441 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.421849 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.421853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.421855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.421924 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.422179 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.422184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.422187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.422357 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:12.422436 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:12.422695 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 16:39:12.422840 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":38}
I20260927 16:39:12.423043 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.414640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46569 (local address 127.0.0.1:58580)
0927 16:39:12.415405 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:12.415418 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:12.415515 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:12.415778 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:12.415781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:12.415976 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:12.416075 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.416081 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.417030 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.417033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.417636 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.417643 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.418287 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.418290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.418400 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.419252 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:12.419270 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:12.419627 (+ 357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:12.421523 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.421526 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.421528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.421757 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.422009 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.422012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.422013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.422093 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.422464 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:12.422467 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:12.422535 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 16:39:12.422644 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 16:39:12.422772 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":70,"threads_started":1}
Sep 27 16:39:12 dist-test-slave-z40b krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527151, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 16:39:12.435060 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.427324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58594 (local address 127.0.98.61:46569)
0927 16:39:12.427457 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:12.427461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:12.427474 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:39:12.427558 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:12.427561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:12.427613 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:12.427692 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:12.428233 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.428715 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.429530 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.429745 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.429844 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.432105 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:12.432109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:12.432111 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:12.432133 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:12.433553 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.434046 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.434048 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.434049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.434114 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.434382 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.434384 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.434385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.434513 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:12.434579 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:12.434902 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 16:39:12.434972 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 16:39:12.435174 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.426897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46569 (local address 127.0.0.1:58594)
0927 16:39:12.427249 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:12.427315 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:12.427433 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:12.427739 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:12.427746 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:12.428049 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:12.428129 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.428134 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.428824 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.428828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.429414 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.429422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.429847 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.429851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.429979 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.430426 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:12.430437 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:12.431926 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:12.433658 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.433666 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.433672 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.433957 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.434212 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.434215 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.434217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.434301 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.434604 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:12.434610 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:12.434729 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 16:39:12.434862 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 16:39:12.435021 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":73,"threads_started":1}
I20260927 16:39:12.436338 6512 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:46569: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 46569
}
attrs {
promote: true
}
}
dest_uuid: "d0dece26d2544ea59c76a1edb6ee1ade"
cas_config_opid_index: -1
I20260927 16:39:12.436864 6512 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: true } }
I20260927 16:39:12.438342 7071 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: true } } }
I20260927 16:39:12.438973 7018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:12.440179 7018 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 1 FOLLOWER]: Refusing update from remote peer d0dece26d2544ea59c76a1edb6ee1ade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 16:39:12.440834 7070 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:39:12.441944 7071 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:46569
I20260927 16:39:12.442368 6976 master.cc:537] Master@127.0.98.61:46569 shutting down...
I20260927 16:39:12.443679 7071 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: true } } }
I20260927 16:39:12.443828 7071 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:12.446233 7018 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: true } } }
I20260927 16:39:12.448575 7078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: true } } }
I20260927 16:39:12.448588 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0dece26d2544ea59c76a1edb6ee1ade. Latest consensus state: current_term: 1 leader_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: true } } }
I20260927 16:39:12.448777 7078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [sys.catalog]: This master's current role is: LEARNER
I20260927 16:39:12.448788 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [sys.catalog]: This master's current role is: LEARNER
I20260927 16:39:12.448957 7077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: true } } }
I20260927 16:39:12.449034 7077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [sys.catalog]: This master's current role is: LEARNER
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260927 16:39:12.452620 7075 mvcc.cc:205] Tried to move back new op lower bound from 7333999212747485184 to 7333999212646420480. Current Snapshot: MvccSnapshot[applied={T|T < 7333999212678127616}]
I20260927 16:39:12.453797 6976 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 1 LEARNER]: Raft consensus shutting down.
I20260927 16:39:12.454051 6976 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 1 LEARNER]: Raft consensus is shut down!
I20260927 16:39:12.454135 6976 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260927 16:39:12.470259 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46845
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41911,127.0.98.61:46569,127.0.98.60:46845
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:39:12.470928 6976 master.cc:552] Master@127.0.98.61:46569 shutdown complete.
I20260927 16:39:12.470975 6976 master_runner.cc:305] Clearing existing system tablet
I20260927 16:39:12.471238 6976 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:12.471290 6976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:12.472549 6976 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 16:39:12.473417 7085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:12.473615 6976 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:12.473788 6976 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 16:39:12.473843 6976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-1/wal
uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed"
format_stamp: "Formatted at 2026-09-27 16:39:12 on dist-test-slave-z40b"
server_key: "8ad427dd6034fe18a07e7d01fa0a3944"
server_key_iv: "6ee2fa703a989734beac7d9a34d0e07a"
server_key_version: "encryptionkey@0"
I20260927 16:39:12.475548 6976 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:39:12.477782 6976 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 16:39:12.477893 6976 log.cc:1196] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 16:39:12.478193 6976 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed: Deleting consensus metadata
I20260927 16:39:12.478379 6976 master_runner.cc:315] Copying system tablet from 127.0.98.62:41911
I20260927 16:39:12.479662 6976 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:41911
I20260927 16:39:12.491123 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.480228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.0.1:33098)
0927 16:39:12.483360 (+ 3132us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:12.483374 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:12.483535 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:12.483791 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:12.483794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:12.483986 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:12.484099 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.484105 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.485088 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.485091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.486046 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.486056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.486635 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.486639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.486745 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.487362 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:12.487375 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:12.487740 (+ 365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:12.489551 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.489554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.489556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.489802 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.490110 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.490113 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.490115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.490200 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.490591 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:12.490594 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:12.490694 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 16:39:12.490797 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 16:39:12.490920 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3066,"thread_start_us":77,"threads_started":1}
I20260927 16:39:12.491142 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.480299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33098 (local address 127.0.98.62:41911)
0927 16:39:12.480412 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:12.480417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:12.483445 (+ 3028us) server_negotiation.cc:407] Connection header received
0927 16:39:12.483629 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:12.483633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:12.483684 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:12.483766 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:12.484206 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.484971 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.486167 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.486524 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.486625 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.487872 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:12.487875 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:12.487878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:12.487902 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:12.489447 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.489904 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.489907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.489908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.490013 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.490293 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.490296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.490298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.490495 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:12.490593 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:12.490879 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 16:39:12.490986 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 16:39:12.492026 6532 tablet_copy_service.cc:140] P d0dece26d2544ea59c76a1edb6ee1ade: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33098)
I20260927 16:39:12.492201 6532 tablet_copy_service.cc:161] P d0dece26d2544ea59c76a1edb6ee1ade: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33098: session id = 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed-00000000000000000000000000000000
I20260927 16:39:12.493741 6532 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 16:39:12.495102 6976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 16:39:12.496995 6976 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed: tablet copy: Starting download of 0 data blocks...
I20260927 16:39:12.497164 6976 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed: tablet copy: Starting download of 1 WAL segments...
I20260927 16:39:12.498642 6976 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 16:39:12.500298 6532 tablet_copy_service.cc:342] P d0dece26d2544ea59c76a1edb6ee1ade: Request end of tablet copy session 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33098
I20260927 16:39:12.500394 6532 tablet_copy_service.cc:433] P d0dece26d2544ea59c76a1edb6ee1ade: ending tablet copy session 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed
I20260927 16:39:12.509357 6976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:12.516422 7092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:12.516577 7095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:12.518663 7093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:12.519065 6976 server_base.cc:1034] running on GCE node
I20260927 16:39:12.519292 6976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:12.519598 6976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:12.520787 6976 hybrid_clock.cc:648] HybridClock initialized: now 1790527152520780 us; error 32 us; skew 500 ppm
Sep 27 16:39:12 dist-test-slave-z40b krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527152, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:12.522272 6976 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 16:39:12.522720 6976 webserver.cc:460] Webserver started at http://127.0.98.61:38707/ using document root <none> and password file <none>
I20260927 16:39:12.522919 6976 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:12.522975 6976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:12.524118 6976 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 16:39:12.525029 7102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:12.525317 6976 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 16:39:12.525404 6976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-1/wal
uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed"
format_stamp: "Formatted at 2026-09-27 16:39:12 on dist-test-slave-z40b"
server_key: "8ad427dd6034fe18a07e7d01fa0a3944"
server_key_iv: "6ee2fa703a989734beac7d9a34d0e07a"
server_key_version: "encryptionkey@0"
I20260927 16:39:12.525484 6976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:12.539032 6976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:12.540988 6976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:12.545949 6976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46569
I20260927 16:39:12.546007 7154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46569 every 8 connection(s)
I20260927 16:39:12.546483 6976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-1/data/info.pb
I20260927 16:39:12.547997 7155 sys_catalog.cc:263] Verifying existing consensus state
I20260927 16:39:12.548765 7155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed: Bootstrap starting.
I20260927 16:39:12.553970 7155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:39:12.554215 7155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed: Bootstrap complete.
I20260927 16:39:12.554683 7155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: true } }
I20260927 16:39:12.554795 7155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed, State: Initialized, Role: LEARNER
I20260927 16:39:12.554920 7155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: true } }
I20260927 16:39:12.555178 7158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: true } } }
I20260927 16:39:12.555347 7158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [sys.catalog]: This master's current role is: LEARNER
I20260927 16:39:12.555362 7155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:39:12.558127 7169 catalog_manager.cc:1259] Loaded cluster ID: 7b929cdf13ac47ff88f0adc0945d6ec6
I20260927 16:39:12.558223 7169 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed: loading cluster ID for follower catalog manager: success
I20260927 16:39:12.559684 7169 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed: acquiring CA information for follower catalog manager: success
I20260927 16:39:12.560695 7169 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 16:39:12.611980 7083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:12.612191 7083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:12.612226 7083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:12.615901 7083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:39:12.615942 7083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:12.615958 7083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:12.615972 7083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:39:12.615985 7083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:39:12.620505 7083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41911,127.0.98.61:46569,127.0.98.60:46845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:12.621323 7083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:12.622181 7083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:12.630477 7175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:12.630477 7174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:12.630824 7177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:12.630796 7083 server_base.cc:1034] running on GCE node
I20260927 16:39:12.631378 7083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:12.632272 7083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:12.633415 7083 hybrid_clock.cc:648] HybridClock initialized: now 1790527152633393 us; error 32 us; skew 500 ppm
Sep 27 16:39:12 dist-test-slave-z40b krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527152, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:12.636057 7083 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 16:39:12.636739 7083 webserver.cc:460] Webserver started at http://127.0.98.60:46541/ using document root <none> and password file <none>
I20260927 16:39:12.637166 7083 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:12.637215 7083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:12.637353 7083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:12.639364 7083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-2/data/instance:
uuid: "7fec907df54a40aeb1da5d546cc4fab8"
format_stamp: "Formatted at 2026-09-27 16:39:12 on dist-test-slave-z40b"
server_key: "d684a14ef2beeb0b6fdd80bf808bcdd4"
server_key_iv: "5e9a49416574140091301cb699747a29"
server_key_version: "encryptionkey@0"
I20260927 16:39:12.639911 7083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-2/wal/instance:
uuid: "7fec907df54a40aeb1da5d546cc4fab8"
format_stamp: "Formatted at 2026-09-27 16:39:12 on dist-test-slave-z40b"
server_key: "d684a14ef2beeb0b6fdd80bf808bcdd4"
server_key_iv: "5e9a49416574140091301cb699747a29"
server_key_version: "encryptionkey@0"
I20260927 16:39:12.643106 7083 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:39:12.645197 7184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:12.646466 7083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:12.646580 7083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-2/wal
uuid: "7fec907df54a40aeb1da5d546cc4fab8"
format_stamp: "Formatted at 2026-09-27 16:39:12 on dist-test-slave-z40b"
server_key: "d684a14ef2beeb0b6fdd80bf808bcdd4"
server_key_iv: "5e9a49416574140091301cb699747a29"
server_key_version: "encryptionkey@0"
I20260927 16:39:12.646678 7083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:12.671729 7083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:12.674253 7083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:12.674427 7083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:12.680934 7083 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46845
I20260927 16:39:12.681207 7236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46845 every 8 connection(s)
I20260927 16:39:12.681851 7083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-2/data/info.pb
I20260927 16:39:12.684293 7237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:12.686409 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 7083
I20260927 16:39:12.687913 7237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:39:12 dist-test-slave-z40b krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527152, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 16:39:12 dist-test-slave-z40b krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 16:39:12.698807 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.689849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33104 (local address 127.0.98.62:41911)
0927 16:39:12.689974 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:12.689977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:12.690007 (+ 30us) server_negotiation.cc:407] Connection header received
0927 16:39:12.690043 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:12.690045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:12.690080 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:12.690145 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:12.690813 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.691581 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.692796 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.693138 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.693224 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.695387 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:12.695393 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:12.695396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:12.695424 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:12.696825 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.697318 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.697321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.697322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.697390 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.697745 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.697748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.697750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.697927 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:12.698005 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:12.698535 (+ 530us) server_negotiation.cc:299] Negotiation successful
0927 16:39:12.698643 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:39:12.699754 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.689339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.0.1:33104)
0927 16:39:12.689662 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:12.689724 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:12.689831 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:12.690230 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:12.690237 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:12.690528 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:12.690668 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.690676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.691688 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.691701 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.692676 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.692685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.693267 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.693273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.693385 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.693778 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:12.693797 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:12.695264 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:12.696926 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.696935 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.696943 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.697191 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.697484 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.697487 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.697489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.697629 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.698049 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:12.698055 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:12.698301 (+ 246us) client_negotiation.cc:769] Sending connection context
0927 16:39:12.698476 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 16:39:12.698795 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":35,"thread_start_us":82,"threads_started":1}
I20260927 16:39:12.699753 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.687785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43914 (local address 127.0.98.60:46845)
0927 16:39:12.688160 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:12.688179 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:12.688210 (+ 31us) server_negotiation.cc:407] Connection header received
0927 16:39:12.688798 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:12.688813 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:12.689129 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:12.689427 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:12.690259 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.691484 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.692548 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.693015 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.693165 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.695841 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:12.695847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:12.695862 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:12.695880 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:12.697561 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.697973 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.697978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.697980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.698051 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.698322 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.698325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.698327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.698618 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:12.698793 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:12.699308 (+ 515us) server_negotiation.cc:299] Negotiation successful
0927 16:39:12.699424 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":258,"thread_start_us":140,"threads_started":1}
I20260927 16:39:12.701110 7237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:39:12 dist-test-slave-z40b krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527152, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 16:39:12.708137 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.701593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46569 (local address 127.0.0.1:58600)
0927 16:39:12.701671 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:12.701684 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:12.701779 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:12.702144 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:12.702145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:12.702259 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:12.702361 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.702365 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.703164 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.703166 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.703738 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.703745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.704137 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.704140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.704201 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.704603 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:12.704614 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:12.705668 (+ 1054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:12.707095 (+ 1427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.707097 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.707098 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.707320 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.707534 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.707536 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.707537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.707598 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.707845 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:12.707847 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:12.707904 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 16:39:12.707970 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 16:39:12.708045 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 16:39:12.708204 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.701714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58600 (local address 127.0.98.61:46569)
0927 16:39:12.701962 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:12.701965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:12.701976 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:39:12.702008 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:12.702010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:12.702044 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:12.702115 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:12.702457 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.703079 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.703842 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.704051 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.704125 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.705785 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:12.705788 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:12.705790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:12.705810 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:12.706993 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.707394 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.707396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.707397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.707462 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.707648 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.707650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.707651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.707781 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:12.707842 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:12.707997 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 16:39:12.708074 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":97,"threads_started":1}
I20260927 16:39:12.708843 7237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46845 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:39:12 dist-test-slave-z40b krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527152, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 16:39:12.715687 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.709252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46845 (local address 127.0.0.1:43928)
0927 16:39:12.709364 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:12.709378 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:12.709467 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:12.709671 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:12.709673 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:12.709815 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:12.709865 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.709869 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.710548 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.710551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.711026 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.711032 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.711530 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.711533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.711597 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.711996 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:12.712009 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:12.713012 (+ 1003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:12.714382 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.714387 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.714389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.714560 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.714811 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.714815 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.714817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.714890 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.715224 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:12.715227 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:12.715299 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 16:39:12.715405 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 16:39:12.715524 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 16:39:12.715704 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.709282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43928 (local address 127.0.98.60:46845)
0927 16:39:12.709365 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:12.709369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:12.709434 (+ 65us) server_negotiation.cc:407] Connection header received
0927 16:39:12.709546 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:12.709550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:12.709581 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:12.709663 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:12.709962 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.710443 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.711137 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.711436 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.711531 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.713138 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:12.713141 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:12.713143 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:12.713163 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:12.714281 (+ 1118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.714647 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.714653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.714655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.714726 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.714974 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.714977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.714979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.715123 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:12.715243 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:12.715445 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 16:39:12.715586 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":32}
I20260927 16:39:12.718464 7237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8: Bootstrap starting.
I20260927 16:39:12.720487 7237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:12.721114 7237 log.cc:824] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:12.722654 7237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8: No bootstrap required, opened a new log
I20260927 16:39:12.725562 7237 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46845 } }
I20260927 16:39:12.725888 7237 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:12.725932 7237 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fec907df54a40aeb1da5d546cc4fab8, State: Initialized, Role: FOLLOWER
I20260927 16:39:12.726325 7237 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46845 } }
I20260927 16:39:12.726778 7243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46845 } } }
I20260927 16:39:12.726958 7243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:39:12.727576 7237 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 16:39:12.732334 7254 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:39:12.732398 7254 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:39:12.734949 7083 master_runner.cc:428] Detected that this master 7fec907df54a40aeb1da5d546cc4fab8 is joining an existing cluster
I20260927 16:39:12.735006 7083 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:46845
I20260927 16:39:12.744963 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.737194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46569 (local address 127.0.0.1:58610)
0927 16:39:12.737731 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:12.737743 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:12.737854 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:12.738319 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:12.738321 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:12.738506 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:12.738590 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.738596 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.739570 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.739573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.740410 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.740418 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.741206 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.741210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.741308 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.741719 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:12.741731 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:12.742032 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:12.743618 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.743622 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.743624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.743808 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.744065 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.744069 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.744071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.744152 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.744492 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:12.744495 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:12.744574 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 16:39:12.744680 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 16:39:12.744794 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":74,"threads_started":1}
I20260927 16:39:12.744957 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.737467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58610 (local address 127.0.98.61:46569)
0927 16:39:12.737567 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:12.737572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:12.737802 (+ 230us) server_negotiation.cc:407] Connection header received
0927 16:39:12.737949 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:12.737952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:12.737995 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:12.738043 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:12.738736 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.739451 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.740529 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.740858 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.741235 (+ 377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.742158 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:12.742161 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:12.742163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:12.742186 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:12.743518 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.743899 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.743904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.743906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.743980 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.744242 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.744247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.744249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.744404 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:12.744502 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:12.744731 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 16:39:12.744835 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 16:39:12.752915 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 6574
I20260927 16:39:12.760191 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.737222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43934 (local address 127.0.98.60:46845)
0927 16:39:12.737318 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:12.737321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:12.737483 (+ 162us) server_negotiation.cc:407] Connection header received
0927 16:39:12.737611 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:12.737616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:12.737648 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:12.737688 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:12.738273 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.739133 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.740271 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.740584 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.740715 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.756205 (+ 15490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:12.756211 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:12.756217 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:12.756247 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:12.757958 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.758402 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.758408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.758413 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.758497 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.759255 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.759261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.759265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.759487 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:12.759579 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:12.759887 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 16:39:12.760021 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":44}
I20260927 16:39:12.760208 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42395
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41911
--webserver_port=41617
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:41911,127.0.98.61:46569
--tserver_master_addrs=127.0.98.62:41911,127.0.98.61:46569,127.0.98.60:46845 with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:39:12.760622 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.736964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46845 (local address 127.0.0.1:43934)
0927 16:39:12.737416 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:12.737427 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:12.737529 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:12.737796 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:12.737798 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:12.738034 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:12.738151 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.738158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.739276 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.739279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.740140 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.740151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.740820 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.740823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.740904 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.755634 (+ 14730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:12.755648 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:12.756046 (+ 398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:12.758068 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.758071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.758073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.758302 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.758591 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.758594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.758596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.759126 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.759614 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:12.759620 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:12.759709 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 16:39:12.760294 (+ 585us) client_negotiation.cc:240] Negotiation successful
0927 16:39:12.760418 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":2,"thread_start_us":81,"threads_started":1}
I20260927 16:39:12.764356 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.736850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.0.1:33106)
0927 16:39:12.737230 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:12.737245 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:12.737377 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:12.737730 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:12.737733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:12.737961 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:12.738088 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.738097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.739084 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.739087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.739959 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.739967 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.740662 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.740665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.740772 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.741275 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:12.741289 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:12.760092 (+ 18803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:12.762733 (+ 2641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.762736 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.762737 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.762970 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.763301 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.763304 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.763306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.763390 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.763871 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:12.763874 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:12.763966 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 16:39:12.764068 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 16:39:12.764185 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":7,"thread_start_us":77,"threads_started":1}
I20260927 16:39:12.764489 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.736956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33106 (local address 127.0.98.62:41911)
0927 16:39:12.737062 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:12.737067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:12.737307 (+ 240us) server_negotiation.cc:407] Connection header received
0927 16:39:12.737488 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:12.737492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:12.737541 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:12.737592 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:12.738215 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.738966 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.740095 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.740437 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.740517 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.760266 (+ 19749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:12.760269 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:12.760273 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:12.760300 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:12.762615 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.763074 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.763080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.763085 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.763188 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.763489 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.763495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.763500 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.763767 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:12.763872 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:12.764155 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 16:39:12.764291 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 16:39:12 dist-test-slave-z40b krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527151, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 16:39:12.777084 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.768184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46845 (local address 127.0.0.1:43950)
0927 16:39:12.768333 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:12.768346 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:12.768417 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:12.768790 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:12.768795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:12.768995 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:12.769121 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.769131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.770126 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.770133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.771192 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.771206 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.771963 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.771967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.772056 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.772494 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:12.772507 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:12.773919 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:12.775758 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.775761 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.775763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.775970 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.776212 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.776215 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.776217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.776287 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.776667 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:12.776670 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:12.776756 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 16:39:12.776838 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 16:39:12.776947 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 16:39:12.777539 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.768224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43950 (local address 127.0.98.60:46845)
0927 16:39:12.768538 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:12.768542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:12.768552 (+ 10us) server_negotiation.cc:407] Connection header received
0927 16:39:12.768604 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:12.768606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:12.768647 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:12.768721 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:12.769261 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.770011 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.771350 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.771724 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.771841 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.774062 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:12.774065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:12.774068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:12.774088 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:12.775640 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.776064 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.776067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.776068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.776131 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.776389 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.776392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.776394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.776560 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:12.776632 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:12.777306 (+ 674us) server_negotiation.cc:299] Negotiation successful
0927 16:39:12.777409 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258}
I20260927 16:39:12.778524 6512 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:46845: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 46845
}
attrs {
promote: true
}
}
dest_uuid: "d0dece26d2544ea59c76a1edb6ee1ade"
cas_config_opid_index: 5
I20260927 16:39:12.779067 6512 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: true } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: true } }
I20260927 16:39:12.780006 7071 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7fec907df54a40aeb1da5d546cc4fab8 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: true } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: true } } }
I20260927 16:39:12.780571 7211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:12.781224 7071 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:46845
I20260927 16:39:12.781399 7071 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: true } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: true } } }
I20260927 16:39:12.781492 7071 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:12.781590 7083 master.cc:537] Master@127.0.98.60:46845 shutting down...
I20260927 16:39:12.781870 7211 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [term 1 FOLLOWER]: Refusing update from remote peer d0dece26d2544ea59c76a1edb6ee1ade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 16:39:12.782372 7070 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 16:39:12.783623 6496 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 16:39:12.784479 6496 consensus_peers.cc:487] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade -> Peer 7fec907df54a40aeb1da5d546cc4fab8 (127.0.98.60:46845): Couldn't send request to peer 7fec907df54a40aeb1da5d546cc4fab8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 16:39:12.786217 7083 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:39:12.786527 7083 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:39:12.786676 7083 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8: stopping tablet replica
I20260927 16:39:12.790324 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.781644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58616 (local address 127.0.98.61:46569)
0927 16:39:12.781796 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:12.781800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:12.785514 (+ 3714us) server_negotiation.cc:407] Connection header received
0927 16:39:12.785926 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:12.785929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:12.785986 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:12.786047 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:12.786372 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.787123 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.788506 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.789090 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.789160 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.789278 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:12.790143 (+ 865us) server_negotiation.cc:299] Negotiation successful
0927 16:39:12.790213 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 16:39:12.790619 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.780173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46569 (local address 127.0.0.1:58616)
0927 16:39:12.785428 (+ 5255us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:12.785441 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:12.785799 (+ 358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:12.786176 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:12.786178 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:12.786193 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:12.786261 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.786266 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.787270 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.787273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.788380 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.788390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.789879 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.789882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.789972 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.790028 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 16:39:12.790453 (+ 425us) client_negotiation.cc:240] Negotiation successful
0927 16:39:12.790517 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5198}
I20260927 16:39:12.791258 7129 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 1 LEARNER]: Refusing update from remote peer d0dece26d2544ea59c76a1edb6ee1ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 16:39:12.791631 7070 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 16:39:12.792534 7083 master.cc:552] Master@127.0.98.60:46845 shutdown complete.
I20260927 16:39:12.792609 7083 master_runner.cc:305] Clearing existing system tablet
I20260927 16:39:12.792523 7129 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7fec907df54a40aeb1da5d546cc4fab8 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: true } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: true } } }
I20260927 16:39:12.792857 7083 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:12.792898 7083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:12.794019 7083 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 16:39:12.794013 7158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0dece26d2544ea59c76a1edb6ee1ade. Latest consensus state: current_term: 1 leader_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: true } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: true } } }
I20260927 16:39:12.794103 7158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [sys.catalog]: This master's current role is: LEARNER
I20260927 16:39:12.794173 7158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: true } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: true } } }
I20260927 16:39:12.794231 7158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [sys.catalog]: This master's current role is: LEARNER
I20260927 16:39:12.794770 7268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:12.794983 7083 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:12.795089 7083 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 16:39:12.795116 7083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-2/wal
uuid: "7fec907df54a40aeb1da5d546cc4fab8"
format_stamp: "Formatted at 2026-09-27 16:39:12 on dist-test-slave-z40b"
server_key: "d684a14ef2beeb0b6fdd80bf808bcdd4"
server_key_iv: "5e9a49416574140091301cb699747a29"
server_key_version: "encryptionkey@0"
I20260927 16:39:12.796880 7083 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:39:12.798847 7083 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 16:39:12.798913 7083 log.cc:1196] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 16:39:12.799119 7083 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8: Deleting consensus metadata
I20260927 16:39:12.799468 7083 master_runner.cc:315] Copying system tablet from 127.0.98.62:41911
I20260927 16:39:12.799966 7070 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade: attempting to promote NON_VOTER 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed to VOTER
I20260927 16:39:12.800352 7070 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: true } }
I20260927 16:39:12.800441 7083 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:41911
I20260927 16:39:12.801599 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.801364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46845 (local address 127.0.0.1:43958)
0927 16:39:12.801484 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:12.801537 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 16:39:12.801820 7129 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 1 LEARNER]: Refusing update from remote peer d0dece26d2544ea59c76a1edb6ee1ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 16:39:12.802155 7072 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 16:39:12.803270 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.803012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46845 (local address 127.0.0.1:43970)
0927 16:39:12.803150 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:12.803192 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260927 16:39:12.803390 6496 consensus_peers.cc:487] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade -> Peer 7fec907df54a40aeb1da5d546cc4fab8 (127.0.98.60:46845): Couldn't send request to peer 7fec907df54a40aeb1da5d546cc4fab8. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 16:39:12.803989 7072 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: true } } }
I20260927 16:39:12.805267 7072 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: true } } }
I20260927 16:39:12.805356 7072 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:12.805457 7072 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: true } } }
I20260927 16:39:12.805534 7072 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:12.806208 7129 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: true } } }
I20260927 16:39:12.808494 7267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: true } } }
I20260927 16:39:12.808591 7267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:39:12.810550 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.800974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33120 (local address 127.0.98.62:41911)
0927 16:39:12.801160 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:12.801164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:12.801755 (+ 591us) server_negotiation.cc:407] Connection header received
0927 16:39:12.801952 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:12.801956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:12.802006 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:12.802054 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:12.803543 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.804287 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.805466 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.805833 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.805909 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.807162 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:12.807165 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:12.807168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:12.807192 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:12.808869 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.809323 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.809326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.809328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.809405 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.809686 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.809689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.809691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.809893 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:12.809962 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:12.810310 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 16:39:12.810414 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260927 16:39:12.810981 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.801354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.0.1:33120)
0927 16:39:12.801661 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:12.801673 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:12.801831 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:12.802174 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:12.802176 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:12.802372 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:12.802498 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.802508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.804423 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.804426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.805334 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.805344 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.806057 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.806060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.806141 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.806603 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:12.806622 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:12.807012 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:12.808980 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.808983 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.808985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.809195 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.809509 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.809513 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.809515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.809594 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.810050 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:12.810053 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:12.810136 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 16:39:12.810672 (+ 536us) client_negotiation.cc:240] Negotiation successful
0927 16:39:12.810811 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":84,"threads_started":1}
I20260927 16:39:12.811703 6532 tablet_copy_service.cc:140] P d0dece26d2544ea59c76a1edb6ee1ade: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7fec907df54a40aeb1da5d546cc4fab8 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33120)
I20260927 16:39:12.811800 6532 tablet_copy_service.cc:161] P d0dece26d2544ea59c76a1edb6ee1ade: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7fec907df54a40aeb1da5d546cc4fab8 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33120: session id = 7fec907df54a40aeb1da5d546cc4fab8-00000000000000000000000000000000
I20260927 16:39:12.812749 6532 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 16:39:12.813787 7083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 16:39:12.815783 7083 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8: tablet copy: Starting download of 0 data blocks...
I20260927 16:39:12.815939 7083 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8: tablet copy: Starting download of 1 WAL segments...
I20260927 16:39:12.817108 7083 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 16:39:12.818496 6532 tablet_copy_service.cc:342] P d0dece26d2544ea59c76a1edb6ee1ade: Request end of tablet copy session 7fec907df54a40aeb1da5d546cc4fab8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33120
I20260927 16:39:12.818583 6532 tablet_copy_service.cc:433] P d0dece26d2544ea59c76a1edb6ee1ade: ending tablet copy session 7fec907df54a40aeb1da5d546cc4fab8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7fec907df54a40aeb1da5d546cc4fab8
I20260927 16:39:12.829563 7083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:12.831682 7275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:12.832257 7278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:12.833175 7083 server_base.cc:1034] running on GCE node
W20260927 16:39:12.835351 7276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:12.835565 7083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:12.835881 7083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:12.837016 7083 hybrid_clock.cc:648] HybridClock initialized: now 1790527152837013 us; error 30 us; skew 500 ppm
Sep 27 16:39:12 dist-test-slave-z40b krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527152, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:12.838522 7083 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 16:39:12.839010 7083 webserver.cc:460] Webserver started at http://127.0.98.60:40791/ using document root <none> and password file <none>
I20260927 16:39:12.839190 7083 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:12.839277 7083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:12.840298 7083 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 16:39:12.841164 7285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:12.841394 7083 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 16:39:12.841467 7083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-2/wal
uuid: "7fec907df54a40aeb1da5d546cc4fab8"
format_stamp: "Formatted at 2026-09-27 16:39:12 on dist-test-slave-z40b"
server_key: "d684a14ef2beeb0b6fdd80bf808bcdd4"
server_key_iv: "5e9a49416574140091301cb699747a29"
server_key_version: "encryptionkey@0"
I20260927 16:39:12.841547 7083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:12.851791 7083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:12.853641 7083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:12.858453 7083 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46845
I20260927 16:39:12.858470 7337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46845 every 8 connection(s)
I20260927 16:39:12.858955 7083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/master-2/data/info.pb
I20260927 16:39:12.860299 7338 sys_catalog.cc:263] Verifying existing consensus state
I20260927 16:39:12.861075 7338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8: Bootstrap starting.
I20260927 16:39:12.869412 7338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:39:12.870846 7338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8: Bootstrap complete.
I20260927 16:39:12.871385 7338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: true } }
I20260927 16:39:12.871528 7338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7fec907df54a40aeb1da5d546cc4fab8, State: Initialized, Role: LEARNER
I20260927 16:39:12.871639 7338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: true } }
I20260927 16:39:12.871927 7341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: true } } }
I20260927 16:39:12.872081 7341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [sys.catalog]: This master's current role is: LEARNER
I20260927 16:39:12.872079 7338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:39:12.875624 7352 catalog_manager.cc:1259] Loaded cluster ID: 7b929cdf13ac47ff88f0adc0945d6ec6
I20260927 16:39:12.875694 7352 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8: loading cluster ID for follower catalog manager: success
I20260927 16:39:12.877097 7352 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8: acquiring CA information for follower catalog manager: success
I20260927 16:39:12.878067 7352 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 16:39:12.914960 7263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:12.915287 7263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:12.915349 7263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:12.920085 7263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:12.920135 7263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:12.920192 7263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:39:12.924878 7263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41617
--tserver_master_addrs=127.0.98.62:41911,127.0.98.61:46569,127.0.98.60:46845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:12.925894 7263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:12.927116 7263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:12.935746 7358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:12.935748 7357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:12.935736 7360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:12.936017 7263 server_base.cc:1034] running on GCE node
I20260927 16:39:12.936318 7263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:12.936865 7263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:12.938014 7263 hybrid_clock.cc:648] HybridClock initialized: now 1790527152937998 us; error 23 us; skew 500 ppm
Sep 27 16:39:12 dist-test-slave-z40b krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527152, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:12.940609 7263 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 16:39:12.941409 7263 webserver.cc:460] Webserver started at http://127.0.98.1:41617/ using document root <none> and password file <none>
I20260927 16:39:12.941915 7263 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:12.942004 7263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:12.945348 7263 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:39:12.947824 7367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:12.948851 7263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 16:39:12.948971 7263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "c7c8dd9decf94b0eb2498082d63112a3"
format_stamp: "Formatted at 2026-09-27 16:39:11 on dist-test-slave-z40b"
server_key: "86436f94d3260fffc8250d4be93d397b"
server_key_iv: "eafc2c41dd2056ec5a07ac946e49d54e"
server_key_version: "encryptionkey@0"
I20260927 16:39:12.949491 7263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:12.969211 7263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:12.972014 7263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:12.972213 7263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:12.972239 6837 heartbeater.cc:503] Master 127.0.98.62:41911 was elected leader, sending a full tablet report...
I20260927 16:39:12.972856 7263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:12.973778 7263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:12.973850 7263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:12.973901 7263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:12.973930 7263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:12.982583 7263 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42395
I20260927 16:39:12.982616 7480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42395 every 8 connection(s)
I20260927 16:39:12.983448 7263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:39:12.987449 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 7263
I20260927 16:39:12.987639 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 6708
Sep 27 16:39:12 dist-test-slave-z40b krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527152, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 16:39:12 dist-test-slave-z40b krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527152, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 16:39:12.995826 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45479
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41911
--webserver_port=45501
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:41911,127.0.98.61:46569
--tserver_master_addrs=127.0.98.62:41911,127.0.98.61:46569,127.0.98.60:46845 with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 16:39:12 dist-test-slave-z40b krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527152, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 16:39:12.998934 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.985075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36107 (local address 127.0.98.61:46569)
0927 16:39:12.985194 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:12.985199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:12.986008 (+ 809us) server_negotiation.cc:407] Connection header received
0927 16:39:12.987009 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:12.987013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:12.987063 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:12.987113 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:12.988757 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.989572 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.990700 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.990934 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.991044 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.993917 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:12.993922 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:12.993925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:12.993951 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:12.995835 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.996404 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.996410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.996415 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.996497 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.997838 (+ 1341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.997843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.997845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.998050 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:12.998142 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:12.998683 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 16:39:12.998774 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:39:12.999168 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.986442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57313 (local address 127.0.98.62:41911)
0927 16:39:12.986563 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:12.986567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:12.986642 (+ 75us) server_negotiation.cc:407] Connection header received
0927 16:39:12.987009 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:12.987013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:12.987062 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:12.987113 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:12.989727 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.990612 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.991967 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.992343 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.992429 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.994904 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:12.994909 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:12.994914 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:12.994944 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:12.997280 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.998012 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.998015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.998017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.998090 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:12.998373 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:12.998376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:12.998377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:12.998553 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:12.998615 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:12.998913 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 16:39:12.999020 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:39:13.000406 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.985386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46569 (local address 127.0.98.1:36107)
0927 16:39:12.985871 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:12.985920 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:12.986823 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:12.987386 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:12.987395 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:12.988403 (+ 1008us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:12.988589 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.988606 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.989714 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.989717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.990568 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.990576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.991072 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.991076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.991339 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.991897 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:12.991919 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:12.993739 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:12.995945 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.995952 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.995965 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.996289 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.997628 (+ 1339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.997630 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.997631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.997742 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.998189 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:12.998194 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:12.998450 (+ 256us) client_negotiation.cc:769] Sending connection context
0927 16:39:12.998627 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 16:39:12.998937 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":36,"spinlock_wait_cycles":27264,"thread_start_us":95,"threads_started":1}
I20260927 16:39:13.001998 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.985386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46845 (local address 127.0.98.1:38933)
0927 16:39:12.985870 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:12.985919 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:12.986800 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:12.987460 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:12.987466 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:12.988276 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:12.988603 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.988614 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.993529 (+ 4915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.993532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.994921 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.994932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.995578 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.995581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.995657 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.996461 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:12.996477 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:12.997775 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:13.000499 (+ 2724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:13.000502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:13.000504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:13.000705 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:13.001007 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:13.001012 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:13.001017 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:13.001103 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:13.001474 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:13.001478 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:13.001567 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 16:39:13.001667 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 16:39:13.001797 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":108,"spinlock_wait_cycles":896,"thread_start_us":158,"threads_started":1}
I20260927 16:39:13.002007 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.985050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38933 (local address 127.0.98.60:46845)
0927 16:39:12.985286 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:12.985290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:12.986333 (+ 1043us) server_negotiation.cc:407] Connection header received
0927 16:39:12.986996 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:12.986999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:12.987057 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:12.987161 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:12.992081 (+ 4920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.993388 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.995034 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.995453 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.995559 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.998065 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:12.998070 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:12.998074 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:12.998104 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:13.000399 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.000808 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.000813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.000815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.000904 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.001180 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.001185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.001187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.001385 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:13.001470 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:13.001728 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 16:39:13.001841 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":87,"threads_started":1}
I20260927 16:39:13.002265 7483 heartbeater.cc:347] Connected to a master server at 127.0.98.61:46569
I20260927 16:39:13.002473 7483 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:13.002571 7482 heartbeater.cc:347] Connected to a master server at 127.0.98.60:46845
I20260927 16:39:13.002640 7482 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:13.002998 7483 heartbeater.cc:511] Master 127.0.98.61:46569 requested a full tablet report, sending...
I20260927 16:39:13.002998 7482 heartbeater.cc:511] Master 127.0.98.60:46845 requested a full tablet report, sending...
I20260927 16:39:13.003177 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:12.986337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.1:57313)
0927 16:39:12.986567 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:12.986581 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:12.986822 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:12.987386 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:12.987395 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:12.987954 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:12.988588 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.988606 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.990761 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.990767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.991841 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:12.991853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:12.992582 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:12.992587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:12.992716 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:12.993180 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:12.993198 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:12.994766 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:12.997438 (+ 2672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.997440 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.997442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.997931 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.998183 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:12.998186 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:12.998188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:12.998274 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:12.998676 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:12.998678 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:12.998749 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 16:39:12.998841 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 16:39:13.002927 (+ 4086us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":3970,"thread_start_us":71,"threads_started":1}
I20260927 16:39:13.003763 7481 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41911
I20260927 16:39:13.003832 7481 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:13.004004 7481 heartbeater.cc:511] Master 127.0.98.62:41911 requested a full tablet report, sending...
I20260927 16:39:13.004659 7302 ts_manager.cc:194] Registered new tserver with Master: c7c8dd9decf94b0eb2498082d63112a3 (127.0.98.1:42395)
I20260927 16:39:13.004657 7119 ts_manager.cc:194] Registered new tserver with Master: c7c8dd9decf94b0eb2498082d63112a3 (127.0.98.1:42395)
I20260927 16:39:13.004760 6512 ts_manager.cc:194] Re-registered known tserver with Master: c7c8dd9decf94b0eb2498082d63112a3 (127.0.98.1:42395)
I20260927 16:39:13.005348 6512 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57313
W20260927 16:39:13.108441 7489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:13.108656 7489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:13.108692 7489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:13.112430 7489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:13.112471 7489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:13.112543 7489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:39:13.117121 7489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45479
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45501
--tserver_master_addrs=127.0.98.62:41911,127.0.98.61:46569,127.0.98.60:46845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:13.117902 7489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:13.118745 7489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:13.128134 7497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:13.128142 7495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:13.128142 7494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:13.128441 7489 server_base.cc:1034] running on GCE node
I20260927 16:39:13.128821 7489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:13.129391 7489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:13.130568 7489 hybrid_clock.cc:648] HybridClock initialized: now 1790527153130558 us; error 27 us; skew 500 ppm
Sep 27 16:39:13 dist-test-slave-z40b krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527153, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:13.133097 7489 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 16:39:13.133872 7489 webserver.cc:460] Webserver started at http://127.0.98.2:45501/ using document root <none> and password file <none>
I20260927 16:39:13.134366 7489 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:13.134444 7489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:13.137859 7489 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:13.139803 7504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:13.140955 7489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:13.141085 7489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "aa1787383203493a93865001e604596f"
format_stamp: "Formatted at 2026-09-27 16:39:11 on dist-test-slave-z40b"
server_key: "ce54e4a2d2da0bf1b8b3d55689e8d60f"
server_key_iv: "4ace39de14582b6a1b3f0de2514aa02f"
server_key_version: "encryptionkey@0"
I20260927 16:39:13.141561 7489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:13.163802 7489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:13.166983 7489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:13.167136 7489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:13.167755 7489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:13.168639 7489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:13.168682 7489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:13.168737 7489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:13.168759 7489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:13.177134 7489 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45479
I20260927 16:39:13.177175 7617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45479 every 8 connection(s)
I20260927 16:39:13.178004 7489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:39:13.182865 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 7489
I20260927 16:39:13.183037 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 6842
I20260927 16:39:13.183168 6971 heartbeater.cc:503] Master 127.0.98.62:41911 was elected leader, sending a full tablet report...
Sep 27 16:39:13 dist-test-slave-z40b krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527153, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 16:39:13 dist-test-slave-z40b krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527153, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 16:39:13.190814 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33697
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41911
--webserver_port=39631
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:41911,127.0.98.61:46569
--tserver_master_addrs=127.0.98.62:41911,127.0.98.61:46569,127.0.98.60:46845 with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 16:39:13 dist-test-slave-z40b krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527153, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 16:39:13.194196 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.179561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35733 (local address 127.0.98.61:46569)
0927 16:39:13.179675 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:13.179679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:13.180592 (+ 913us) server_negotiation.cc:407] Connection header received
0927 16:39:13.181646 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:13.181649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:13.181698 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:13.181748 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:13.182945 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.183714 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.185474 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.185921 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.186042 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.189769 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:13.189773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:13.189776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:13.189800 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:13.192294 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.192727 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.192733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.192734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.192817 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.193124 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.193127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.193129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.193343 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:13.193444 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:13.193899 (+ 455us) server_negotiation.cc:299] Negotiation successful
0927 16:39:13.194012 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 16:39:13.195401 7622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.179935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46569 (local address 127.0.98.2:35733)
0927 16:39:13.180426 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:13.180473 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:13.181446 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:13.181926 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:13.181933 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:13.182337 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:13.182798 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:13.182806 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.184062 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.184065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:13.185354 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:13.185363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.186066 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.186070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:13.186269 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.186901 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:13.186924 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:13.189610 (+ 2686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:13.192400 (+ 2790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:13.192405 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:13.192407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:13.192623 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:13.192920 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:13.192926 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:13.192930 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:13.193021 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:13.193456 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:13.193462 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:13.193636 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 16:39:13.194524 (+ 888us) client_negotiation.cc:240] Negotiation successful
0927 16:39:13.194664 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":106,"spinlock_wait_cycles":40832,"thread_start_us":552,"threads_started":1}
I20260927 16:39:13.195649 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.181033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42413 (local address 127.0.98.62:41911)
0927 16:39:13.181160 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:13.181165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:13.181223 (+ 58us) server_negotiation.cc:407] Connection header received
0927 16:39:13.181646 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:13.181649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:13.181698 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:13.181756 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:13.184004 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.184759 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.185784 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.186056 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.186152 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.189764 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:13.189768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:13.189770 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:13.189791 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:13.191783 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.192452 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.192457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.192461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.192550 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.192957 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.192960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.192962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.193182 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:13.193284 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:13.195347 (+ 2063us) server_negotiation.cc:299] Negotiation successful
0927 16:39:13.195468 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:39:13.195660 7624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.180886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.2:42413)
0927 16:39:13.181149 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:13.181162 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:13.181446 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:13.181926 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:13.181933 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:13.182307 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:13.183884 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:13.183891 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.184890 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.184893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:13.185654 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:13.185661 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.186277 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.186281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:13.186359 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.186985 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:13.187001 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:13.189603 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:13.191913 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:13.191922 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:13.191937 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:13.192332 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:13.192654 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:13.192658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:13.192660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:13.192853 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:13.193337 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:13.193343 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:13.193637 (+ 294us) client_negotiation.cc:769] Sending connection context
0927 16:39:13.193819 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 16:39:13.194363 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":1259,"spinlock_wait_cycles":14720,"thread_start_us":113,"threads_started":1}
I20260927 16:39:13.196853 7620 heartbeater.cc:347] Connected to a master server at 127.0.98.61:46569
I20260927 16:39:13.196924 7618 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41911
I20260927 16:39:13.197054 7618 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:13.197054 7620 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:13.197546 7618 heartbeater.cc:511] Master 127.0.98.62:41911 requested a full tablet report, sending...
I20260927 16:39:13.197543 7620 heartbeater.cc:511] Master 127.0.98.61:46569 requested a full tablet report, sending...
I20260927 16:39:13.198522 7623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.179936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46845 (local address 127.0.98.2:60413)
0927 16:39:13.180742 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:13.180757 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:13.181445 (+ 688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:13.181924 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:13.181931 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:13.182303 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:13.182793 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:13.182805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.189451 (+ 6646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.189454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:13.190255 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:13.190262 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.190840 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.190843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:13.190979 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.191969 (+ 990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:13.191988 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:13.193502 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:13.195524 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:13.195529 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:13.195531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:13.195783 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:13.196464 (+ 681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:13.196467 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:13.196470 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:13.196556 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:13.198061 (+ 1505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:13.198063 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:13.198153 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 16:39:13.198257 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 16:39:13.198376 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":77,"thread_start_us":156,"threads_started":1}
I20260927 16:39:13.198639 6512 ts_manager.cc:194] Re-registered known tserver with Master: aa1787383203493a93865001e604596f (127.0.98.2:45479)
I20260927 16:39:13.198650 7119 ts_manager.cc:194] Registered new tserver with Master: aa1787383203493a93865001e604596f (127.0.98.2:45479)
I20260927 16:39:13.199218 6512 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42413
I20260927 16:39:13.199416 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.179572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60413 (local address 127.0.98.60:46845)
0927 16:39:13.179687 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:13.179691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:13.180834 (+ 1143us) server_negotiation.cc:407] Connection header received
0927 16:39:13.181623 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:13.181628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:13.181682 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:13.181774 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:13.182925 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.183638 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.190369 (+ 6731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.190714 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.190820 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.193640 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:13.193643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:13.193646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:13.193671 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:13.195395 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.196272 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.196279 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.196280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.196360 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.196657 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.196663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.196665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.196888 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:13.198078 (+ 1190us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:13.199050 (+ 972us) server_negotiation.cc:299] Negotiation successful
0927 16:39:13.199211 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:39:13.200202 7619 heartbeater.cc:347] Connected to a master server at 127.0.98.60:46845
I20260927 16:39:13.200285 7619 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:13.200577 7619 heartbeater.cc:511] Master 127.0.98.60:46845 requested a full tablet report, sending...
I20260927 16:39:13.201248 7302 ts_manager.cc:194] Registered new tserver with Master: aa1787383203493a93865001e604596f (127.0.98.2:45479)
W20260927 16:39:13.301687 7625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:13.301923 7625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:13.301985 7625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:13.305887 7625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:13.305955 7625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:13.306046 7625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:39:13.310818 7625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33697
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39631
--tserver_master_addrs=127.0.98.62:41911,127.0.98.61:46569,127.0.98.60:46845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:13.311779 7625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:13.312618 7625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:13.321221 7633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:13.321171 7630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:13.321444 7625 server_base.cc:1034] running on GCE node
W20260927 16:39:13.321178 7631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:13.322100 7625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:13.322816 7625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:13.323982 7625 hybrid_clock.cc:648] HybridClock initialized: now 1790527153323971 us; error 29 us; skew 500 ppm
I20260927 16:39:13.325944 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.322004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43972 (local address 127.0.98.60:46845)
0927 16:39:13.322118 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:13.322124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:13.322207 (+ 83us) server_negotiation.cc:407] Connection header received
0927 16:39:13.322329 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:13.322333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:13.322415 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:13.322517 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:13.322744 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.323483 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.324594 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.325303 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.325409 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.325576 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:13.325729 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 16:39:13.325816 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:39:13.325959 7635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.321864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46845 (local address 127.0.0.1:43972)
0927 16:39:13.322098 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:13.322112 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:13.322233 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:13.322531 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:13.322533 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:13.322546 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:13.322636 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:13.322640 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.323595 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.323598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:13.324463 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:13.324472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.325474 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.325478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:13.325561 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.325611 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 16:39:13.325693 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 16:39:13.325817 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":68,"threads_started":1}
I20260927 16:39:13.326762 7309 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
Sep 27 16:39:13 dist-test-slave-z40b krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527153, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:13.327133 7341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0dece26d2544ea59c76a1edb6ee1ade. Latest consensus state: current_term: 1 leader_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: true } } }
I20260927 16:39:13.327317 7341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [sys.catalog]: This master's current role is: LEARNER
I20260927 16:39:13.327569 7625 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 16:39:13.328446 7625 webserver.cc:460] Webserver started at http://127.0.98.3:39631/ using document root <none> and password file <none>
I20260927 16:39:13.328909 7625 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:13.328974 7625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:13.332427 7625 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:39:13.334281 7641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:13.335467 7625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:39:13.335551 7625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "b1f3d8ccb5e64924a7327a038fed3756"
format_stamp: "Formatted at 2026-09-27 16:39:12 on dist-test-slave-z40b"
server_key: "f2a44b65eda2d9259ccbd7db172cd8d4"
server_key_iv: "9e951118032293981070ecd91b7d76ae"
server_key_version: "encryptionkey@0"
I20260927 16:39:13.336021 7625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:13.363117 7625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:13.366279 7625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:13.366425 7625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:13.366923 7625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:13.367820 7625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:13.367861 7625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:13.367895 7625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:13.367916 7625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:13.377319 7625 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33697
I20260927 16:39:13.377342 7754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33697 every 8 connection(s)
I20260927 16:39:13.378211 7625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:39:13.381049 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 7625
Sep 27 16:39:13 dist-test-slave-z40b krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527153, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 16:39:13 dist-test-slave-z40b krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527153, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 16:39:13 dist-test-slave-z40b krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527153, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 16:39:13.395146 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.382823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33134 (local address 127.0.98.62:41911)
0927 16:39:13.383490 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:13.383494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:13.383860 (+ 366us) server_negotiation.cc:407] Connection header received
0927 16:39:13.383959 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:13.383961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:13.383996 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:13.384093 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:13.384992 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.385803 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.386923 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.387275 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.387346 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.389168 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:13.389172 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:13.389175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:13.389198 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:13.392880 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.393365 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.393369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.393372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.393466 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.393794 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.393800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.393804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.394012 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:13.394146 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:13.394785 (+ 639us) server_negotiation.cc:299] Negotiation successful
0927 16:39:13.394916 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":61,"threads_started":1}
I20260927 16:39:13.395972 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.379929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33901 (local address 127.0.98.60:46845)
0927 16:39:13.380086 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:13.380090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:13.381636 (+ 1546us) server_negotiation.cc:407] Connection header received
0927 16:39:13.382290 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:13.382293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:13.382343 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:13.382395 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:13.386642 (+ 4247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.387434 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.388601 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.388910 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.388978 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.391404 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:13.391408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:13.391426 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:13.391453 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:13.393688 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.394172 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.394175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.394177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.394244 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.394526 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.394529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.394531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.394723 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:13.394781 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:13.395734 (+ 953us) server_negotiation.cc:299] Negotiation successful
0927 16:39:13.395833 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 16:39:13.396423 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.381662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54513 (local address 127.0.98.61:46569)
0927 16:39:13.382692 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:13.382696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:13.382712 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:39:13.382767 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:13.382770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:13.382814 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:13.382895 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:13.384087 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.384898 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.386593 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.386918 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.387003 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.390931 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:13.390937 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:13.390943 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:13.390969 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:13.393384 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.393974 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.393979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.393982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.394054 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.394426 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.394429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.394431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.394614 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:13.394692 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:13.396181 (+ 1489us) server_negotiation.cc:299] Negotiation successful
0927 16:39:13.396283 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":947}
I20260927 16:39:13.397120 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.380303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46845 (local address 127.0.98.3:33901)
0927 16:39:13.381553 (+ 1250us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:13.381566 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:13.382155 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:13.382520 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:13.382523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:13.384378 (+ 1855us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:13.384479 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:13.384487 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.387560 (+ 3073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.387565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:13.388468 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:13.388477 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.389115 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.389118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:13.389202 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.389652 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:13.389668 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:13.391238 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:13.393826 (+ 2588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:13.393830 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:13.393832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:13.394070 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:13.394349 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:13.394354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:13.394359 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:13.394433 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:13.394843 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:13.394845 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:13.395041 (+ 196us) client_negotiation.cc:769] Sending connection context
0927 16:39:13.396272 (+ 1231us) client_negotiation.cc:240] Negotiation successful
0927 16:39:13.396379 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1156,"mutex_wait_us":23,"spinlock_wait_cycles":22144,"thread_start_us":112,"threads_started":1}
I20260927 16:39:13.397426 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.381284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46569 (local address 127.0.98.3:54513)
0927 16:39:13.381840 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:13.381853 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:13.381932 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:13.383019 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:13.383022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:13.383316 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:13.383943 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:13.383964 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.385050 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.385053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:13.386465 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:13.386475 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.387135 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.387138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:13.387345 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.388149 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:13.388172 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:13.390751 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:13.393512 (+ 2761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:13.393517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:13.393531 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:13.393872 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:13.394152 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:13.394158 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:13.394160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:13.394329 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:13.394742 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:13.394750 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:13.395041 (+ 291us) client_negotiation.cc:769] Sending connection context
0927 16:39:13.395324 (+ 283us) client_negotiation.cc:240] Negotiation successful
0927 16:39:13.395608 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":51,"spinlock_wait_cycles":7424,"thread_start_us":80,"threads_started":1}
I20260927 16:39:13.398386 7757 heartbeater.cc:347] Connected to a master server at 127.0.98.61:46569
I20260927 16:39:13.398612 7757 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:13.399123 7757 heartbeater.cc:511] Master 127.0.98.61:46569 requested a full tablet report, sending...
I20260927 16:39:13.400144 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.379947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40169 (local address 127.0.98.62:41911)
0927 16:39:13.381535 (+ 1588us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:13.381540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:13.381556 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:39:13.382108 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:13.382112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:13.382167 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:13.382254 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:13.384604 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.385350 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.389443 (+ 4093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.389763 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.389833 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.393889 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:13.393892 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:13.393895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:13.393913 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:13.395526 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.395975 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.395980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.395985 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.396064 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.399304 (+ 3240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.399308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.399310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.399489 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:13.399566 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:13.399893 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 16:39:13.399994 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1507}
I20260927 16:39:13.400200 7119 ts_manager.cc:194] Registered new tserver with Master: b1f3d8ccb5e64924a7327a038fed3756 (127.0.98.3:33697)
I20260927 16:39:13.400601 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.380303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.3:40169)
0927 16:39:13.380818 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:13.380880 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:13.381888 (+ 1008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:13.382447 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:13.382457 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:13.384141 (+ 1684us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:13.384257 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:13.384264 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.385491 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.385494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:13.386494 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:13.386509 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.389981 (+ 3472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.389984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:13.390080 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.390832 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:13.390849 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:13.393025 (+ 2176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:13.395646 (+ 2621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:13.395649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:13.395651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:13.395866 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:13.396182 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:13.396185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:13.396186 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:13.396281 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:13.399623 (+ 3342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:13.399626 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:13.399718 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 16:39:13.400308 (+ 590us) client_negotiation.cc:240] Negotiation successful
0927 16:39:13.400449 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":9,"thread_start_us":661,"threads_started":1}
I20260927 16:39:13.400868 7756 heartbeater.cc:347] Connected to a master server at 127.0.98.60:46845
I20260927 16:39:13.400949 7756 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:13.401104 7755 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41911
I20260927 16:39:13.401182 7755 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:13.401363 7755 heartbeater.cc:511] Master 127.0.98.62:41911 requested a full tablet report, sending...
I20260927 16:39:13.401949 6511 ts_manager.cc:194] Re-registered known tserver with Master: b1f3d8ccb5e64924a7327a038fed3756 (127.0.98.3:33697)
I20260927 16:39:13.402549 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.383286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58628 (local address 127.0.98.61:46569)
0927 16:39:13.384080 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:13.384085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:13.384533 (+ 448us) server_negotiation.cc:407] Connection header received
0927 16:39:13.384789 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:13.384792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:13.384833 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:13.384887 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:13.388686 (+ 3799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.389397 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.390472 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.390814 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.390893 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.396430 (+ 5537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:13.396435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:13.396441 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:13.396466 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:13.398657 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.401167 (+ 2510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.401170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.401172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.401268 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.401585 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.401592 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.401596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.401796 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:13.401907 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:13.402308 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 16:39:13.402413 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":81,"threads_started":1}
I20260927 16:39:13.402586 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.382900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43986 (local address 127.0.98.60:46845)
0927 16:39:13.383608 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:13.383611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:13.384332 (+ 721us) server_negotiation.cc:407] Connection header received
0927 16:39:13.384998 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:13.385001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:13.385043 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:13.385094 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:13.386817 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.387535 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.390314 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.390626 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.390689 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.398918 (+ 8229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:13.398922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:13.398924 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:13.398946 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:13.400623 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.401333 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.401336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.401338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.401416 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.401718 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.401721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.401723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.401901 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:13.402007 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:13.402339 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 16:39:13.402443 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":73,"threads_started":1}
I20260927 16:39:13.402653 6511 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40169
I20260927 16:39:13.399358 6512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 16:39:13.404273 7756 heartbeater.cc:511] Master 127.0.98.60:46845 requested a full tablet report, sending...
W20260927 16:39:13.404397 6512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:39:13.404889 7302 ts_manager.cc:194] Registered new tserver with Master: b1f3d8ccb5e64924a7327a038fed3756 (127.0.98.3:33697)
I20260927 16:39:13.406710 7070 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: true } } }
I20260927 16:39:13.406850 7070 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:13.410112 7775 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade: attempting to promote NON_VOTER 7fec907df54a40aeb1da5d546cc4fab8 to VOTER
I20260927 16:39:13.410468 7775 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: false } }
I20260927 16:39:13.411525 7309 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [term 1 LEARNER]: Refusing update from remote peer d0dece26d2544ea59c76a1edb6ee1ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 16:39:13.411552 7129 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 1 FOLLOWER]: Refusing update from remote peer d0dece26d2544ea59c76a1edb6ee1ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 16:39:13.411937 7775 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 16:39:13.412123 7070 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 16:39:13.413503 7775 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 7fec907df54a40aeb1da5d546cc4fab8 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: false } } }
I20260927 16:39:13.414126 7129 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 7fec907df54a40aeb1da5d546cc4fab8 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: false } } }
I20260927 16:39:13.414299 7309 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 7fec907df54a40aeb1da5d546cc4fab8 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: false } } }
I20260927 16:39:13.415021 7775 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: false } } }
I20260927 16:39:13.415120 7775 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:13.415297 7775 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: false } } }
I20260927 16:39:13.415372 7775 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:13.415462 7775 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: false } } }
I20260927 16:39:13.415531 7775 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:13.416154 7341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: false } } }
I20260927 16:39:13.416173 7783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: false } } }
I20260927 16:39:13.416278 7783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:39:13.416273 7341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:39:13.422396 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.418216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38022 (local address 127.0.98.3:33697)
0927 16:39:13.418488 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:13.418498 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:13.418515 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:39:13.418570 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:13.418576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:13.418746 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:13.418890 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:13.419055 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.420090 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.421375 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.421866 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.421978 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.422103 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:13.422197 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 16:39:13.422281 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":90,"threads_started":1}
I20260927 16:39:13.422516 7635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.417028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45479 (local address 127.0.0.1:55458)
0927 16:39:13.417157 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:13.417171 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:13.417235 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:13.418595 (+ 1360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:13.418597 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:13.418613 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:13.418685 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:13.418690 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.419781 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.419784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:13.421068 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:13.421078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.422100 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.422104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:13.422190 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.422217 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 16:39:13.422265 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 16:39:13.422343 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":82}
I20260927 16:39:13.422363 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.417231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33697 (local address 127.0.0.1:38022)
0927 16:39:13.417422 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:13.417433 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:13.417515 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:13.418869 (+ 1354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:13.418871 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:13.418886 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:13.418970 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:13.418975 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.420193 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.420197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:13.421256 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:13.421264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.421978 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.421982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:13.422075 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.422095 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 16:39:13.422172 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 16:39:13.422239 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":26,"thread_start_us":82,"threads_started":1}
I20260927 16:39:13.422904 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.417788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55458 (local address 127.0.98.2:45479)
0927 16:39:13.418083 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:13.418111 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:13.418132 (+ 21us) server_negotiation.cc:407] Connection header received
0927 16:39:13.418203 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:13.418208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:13.418437 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:13.418629 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:13.418779 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.419652 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.421200 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.421966 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.422430 (+ 464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.422588 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:13.422646 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 16:39:13.422757 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":123,"threads_started":1}
I20260927 16:39:13.423130 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.417516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42395 (local address 127.0.0.1:44518)
0927 16:39:13.417707 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:13.417719 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:13.417797 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:13.418405 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:13.418408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:13.418440 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:13.418528 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:13.418535 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.420060 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.420064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:13.421309 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:13.421316 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.422337 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.422340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:13.422425 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.422680 (+ 255us) client_negotiation.cc:769] Sending connection context
0927 16:39:13.422797 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 16:39:13.422862 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":77,"thread_start_us":72,"threads_started":1}
I20260927 16:39:13.425321 7689 tablet_service.cc:1467] Processing CreateTablet for tablet fe2d878f97964cd4bce0ca9c0478ea8b (DEFAULT_TABLE table=test-table [id=703cfe557f504d598c45263127280a3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:39:13.425334 7552 tablet_service.cc:1467] Processing CreateTablet for tablet fe2d878f97964cd4bce0ca9c0478ea8b (DEFAULT_TABLE table=test-table [id=703cfe557f504d598c45263127280a3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:39:13.426309 7552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe2d878f97964cd4bce0ca9c0478ea8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:13.426333 7689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe2d878f97964cd4bce0ca9c0478ea8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:13.431645 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.417663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44518 (local address 127.0.98.1:42395)
0927 16:39:13.417939 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:13.417949 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:13.417968 (+ 19us) server_negotiation.cc:407] Connection header received
0927 16:39:13.418033 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:13.418038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:13.418247 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:13.418660 (+ 413us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:13.418794 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.419931 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.421431 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.422194 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.423361 (+ 1167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.431337 (+ 7976us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:13.431395 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 16:39:13.431500 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":95,"threads_started":1}
I20260927 16:39:13.432726 7789 tablet_bootstrap.cc:492] T fe2d878f97964cd4bce0ca9c0478ea8b P aa1787383203493a93865001e604596f: Bootstrap starting.
I20260927 16:39:13.432837 7790 tablet_bootstrap.cc:492] T fe2d878f97964cd4bce0ca9c0478ea8b P b1f3d8ccb5e64924a7327a038fed3756: Bootstrap starting.
I20260927 16:39:13.434648 7789 tablet_bootstrap.cc:654] T fe2d878f97964cd4bce0ca9c0478ea8b P aa1787383203493a93865001e604596f: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:13.434827 7415 tablet_service.cc:1467] Processing CreateTablet for tablet fe2d878f97964cd4bce0ca9c0478ea8b (DEFAULT_TABLE table=test-table [id=703cfe557f504d598c45263127280a3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:39:13.435302 7790 tablet_bootstrap.cc:654] T fe2d878f97964cd4bce0ca9c0478ea8b P b1f3d8ccb5e64924a7327a038fed3756: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:13.435442 7789 log.cc:824] T fe2d878f97964cd4bce0ca9c0478ea8b P aa1787383203493a93865001e604596f: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:13.435902 7415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe2d878f97964cd4bce0ca9c0478ea8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:13.436046 7790 log.cc:824] T fe2d878f97964cd4bce0ca9c0478ea8b P b1f3d8ccb5e64924a7327a038fed3756: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:13.437598 7790 tablet_bootstrap.cc:492] T fe2d878f97964cd4bce0ca9c0478ea8b P b1f3d8ccb5e64924a7327a038fed3756: No bootstrap required, opened a new log
I20260927 16:39:13.437637 7789 tablet_bootstrap.cc:492] T fe2d878f97964cd4bce0ca9c0478ea8b P aa1787383203493a93865001e604596f: No bootstrap required, opened a new log
I20260927 16:39:13.437775 7790 ts_tablet_manager.cc:1397] T fe2d878f97964cd4bce0ca9c0478ea8b P b1f3d8ccb5e64924a7327a038fed3756: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260927 16:39:13.437803 7789 ts_tablet_manager.cc:1397] T fe2d878f97964cd4bce0ca9c0478ea8b P aa1787383203493a93865001e604596f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 16:39:13.441555 7790 raft_consensus.cc:348] T fe2d878f97964cd4bce0ca9c0478ea8b P b1f3d8ccb5e64924a7327a038fed3756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1787383203493a93865001e604596f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45479 } } peers { permanent_uuid: "b1f3d8ccb5e64924a7327a038fed3756" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33697 } } peers { permanent_uuid: "c7c8dd9decf94b0eb2498082d63112a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42395 } }
I20260927 16:39:13.441805 7790 raft_consensus.cc:374] T fe2d878f97964cd4bce0ca9c0478ea8b P b1f3d8ccb5e64924a7327a038fed3756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:13.441865 7790 raft_consensus.cc:729] T fe2d878f97964cd4bce0ca9c0478ea8b P b1f3d8ccb5e64924a7327a038fed3756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1f3d8ccb5e64924a7327a038fed3756, State: Initialized, Role: FOLLOWER
I20260927 16:39:13.442142 7789 raft_consensus.cc:348] T fe2d878f97964cd4bce0ca9c0478ea8b P aa1787383203493a93865001e604596f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1787383203493a93865001e604596f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45479 } } peers { permanent_uuid: "b1f3d8ccb5e64924a7327a038fed3756" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33697 } } peers { permanent_uuid: "c7c8dd9decf94b0eb2498082d63112a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42395 } }
I20260927 16:39:13.442287 7790 consensus_queue.cc:260] T fe2d878f97964cd4bce0ca9c0478ea8b P b1f3d8ccb5e64924a7327a038fed3756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1787383203493a93865001e604596f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45479 } } peers { permanent_uuid: "b1f3d8ccb5e64924a7327a038fed3756" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33697 } } peers { permanent_uuid: "c7c8dd9decf94b0eb2498082d63112a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42395 } }
I20260927 16:39:13.442389 7789 raft_consensus.cc:374] T fe2d878f97964cd4bce0ca9c0478ea8b P aa1787383203493a93865001e604596f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:13.442445 7789 raft_consensus.cc:729] T fe2d878f97964cd4bce0ca9c0478ea8b P aa1787383203493a93865001e604596f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa1787383203493a93865001e604596f, State: Initialized, Role: FOLLOWER
I20260927 16:39:13.442857 7789 consensus_queue.cc:260] T fe2d878f97964cd4bce0ca9c0478ea8b P aa1787383203493a93865001e604596f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1787383203493a93865001e604596f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45479 } } peers { permanent_uuid: "b1f3d8ccb5e64924a7327a038fed3756" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33697 } } peers { permanent_uuid: "c7c8dd9decf94b0eb2498082d63112a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42395 } }
I20260927 16:39:13.443200 7790 ts_tablet_manager.cc:1428] T fe2d878f97964cd4bce0ca9c0478ea8b P b1f3d8ccb5e64924a7327a038fed3756: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260927 16:39:13.443533 7618 heartbeater.cc:503] Master 127.0.98.62:41911 was elected leader, sending a full tablet report...
I20260927 16:39:13.443759 7789 ts_tablet_manager.cc:1428] T fe2d878f97964cd4bce0ca9c0478ea8b P aa1787383203493a93865001e604596f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 16:39:13.444406 7755 heartbeater.cc:503] Master 127.0.98.62:41911 was elected leader, sending a full tablet report...
I20260927 16:39:13.448451 7795 tablet_bootstrap.cc:492] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3: Bootstrap starting.
I20260927 16:39:13.450371 7795 tablet_bootstrap.cc:654] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:13.451027 7795 log.cc:824] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:13.452611 7795 tablet_bootstrap.cc:492] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3: No bootstrap required, opened a new log
I20260927 16:39:13.452847 7795 ts_tablet_manager.cc:1397] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 16:39:13.456763 7795 raft_consensus.cc:348] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1787383203493a93865001e604596f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45479 } } peers { permanent_uuid: "b1f3d8ccb5e64924a7327a038fed3756" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33697 } } peers { permanent_uuid: "c7c8dd9decf94b0eb2498082d63112a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42395 } }
I20260927 16:39:13.456983 7795 raft_consensus.cc:374] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:13.457036 7795 raft_consensus.cc:729] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7c8dd9decf94b0eb2498082d63112a3, State: Initialized, Role: FOLLOWER
I20260927 16:39:13.457406 7795 consensus_queue.cc:260] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1787383203493a93865001e604596f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45479 } } peers { permanent_uuid: "b1f3d8ccb5e64924a7327a038fed3756" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33697 } } peers { permanent_uuid: "c7c8dd9decf94b0eb2498082d63112a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42395 } }
I20260927 16:39:13.457930 7481 heartbeater.cc:503] Master 127.0.98.62:41911 was elected leader, sending a full tablet report...
I20260927 16:39:13.458181 7795 ts_tablet_manager.cc:1428] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 16:39:13.458457 7797 raft_consensus.cc:482] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:39:13.458809 7797 raft_consensus.cc:504] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1787383203493a93865001e604596f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45479 } } peers { permanent_uuid: "b1f3d8ccb5e64924a7327a038fed3756" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33697 } } peers { permanent_uuid: "c7c8dd9decf94b0eb2498082d63112a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42395 } }
I20260927 16:39:13.460553 7797 leader_election.cc:288] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa1787383203493a93865001e604596f (127.0.98.2:45479), b1f3d8ccb5e64924a7327a038fed3756 (127.0.98.3:33697)
I20260927 16:39:13.463970 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.460688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55753 (local address 127.0.98.2:45479)
0927 16:39:13.460788 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:13.460794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:13.460811 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:39:13.460846 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:13.460848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:13.460905 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:13.460982 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:13.461283 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.462041 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.462996 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.463455 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.463567 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.463621 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:13.463799 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 16:39:13.463862 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 16:39:13.463980 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.460488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45479 (local address 127.0.98.1:55753)
0927 16:39:13.460612 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:13.460628 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:13.460708 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:13.461010 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:13.461014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:13.461032 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:13.461145 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:13.461151 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.462142 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.462147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:13.462876 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:13.462884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.463566 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.463569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:13.463672 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.463697 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 16:39:13.463787 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 16:39:13.463852 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":1}
I20260927 16:39:13.464071 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.460858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33517 (local address 127.0.98.3:33697)
0927 16:39:13.460990 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:13.460994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:13.461190 (+ 196us) server_negotiation.cc:407] Connection header received
0927 16:39:13.461297 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:13.461300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:13.461360 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:13.461448 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:13.461686 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.462223 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.463143 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.463557 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.463676 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.463729 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:13.463902 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 16:39:13.463962 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:39:13.464092 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.460528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33697 (local address 127.0.98.1:33517)
0927 16:39:13.461122 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:13.461132 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:13.461221 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:13.461454 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:13.461457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:13.461471 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:13.461574 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:13.461579 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.462325 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.462330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:13.463029 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:13.463036 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.463676 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.463679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:13.463782 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.463806 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 16:39:13.463889 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 16:39:13.463954 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":493}
I20260927 16:39:13.464570 7709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2d878f97964cd4bce0ca9c0478ea8b" candidate_uuid: "c7c8dd9decf94b0eb2498082d63112a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1f3d8ccb5e64924a7327a038fed3756" is_pre_election: true
I20260927 16:39:13.464823 7709 raft_consensus.cc:2454] T fe2d878f97964cd4bce0ca9c0478ea8b P b1f3d8ccb5e64924a7327a038fed3756 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7c8dd9decf94b0eb2498082d63112a3 in term 0.
I20260927 16:39:13.464941 7572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2d878f97964cd4bce0ca9c0478ea8b" candidate_uuid: "c7c8dd9decf94b0eb2498082d63112a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa1787383203493a93865001e604596f" is_pre_election: true
I20260927 16:39:13.465173 7572 raft_consensus.cc:2454] T fe2d878f97964cd4bce0ca9c0478ea8b P aa1787383203493a93865001e604596f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7c8dd9decf94b0eb2498082d63112a3 in term 0.
I20260927 16:39:13.465281 7369 leader_election.cc:302] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1f3d8ccb5e64924a7327a038fed3756, c7c8dd9decf94b0eb2498082d63112a3; no voters:
I20260927 16:39:13.465551 7797 raft_consensus.cc:2781] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:39:13.465698 7797 raft_consensus.cc:482] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:39:13.465749 7797 raft_consensus.cc:3037] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:13.466809 7797 raft_consensus.cc:504] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1787383203493a93865001e604596f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45479 } } peers { permanent_uuid: "b1f3d8ccb5e64924a7327a038fed3756" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33697 } } peers { permanent_uuid: "c7c8dd9decf94b0eb2498082d63112a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42395 } }
I20260927 16:39:13.467139 7797 leader_election.cc:288] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3 [CANDIDATE]: Term 1 election: Requested vote from peers aa1787383203493a93865001e604596f (127.0.98.2:45479), b1f3d8ccb5e64924a7327a038fed3756 (127.0.98.3:33697)
I20260927 16:39:13.467374 7572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2d878f97964cd4bce0ca9c0478ea8b" candidate_uuid: "c7c8dd9decf94b0eb2498082d63112a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa1787383203493a93865001e604596f"
I20260927 16:39:13.467527 7572 raft_consensus.cc:3037] T fe2d878f97964cd4bce0ca9c0478ea8b P aa1787383203493a93865001e604596f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:13.467547 7709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2d878f97964cd4bce0ca9c0478ea8b" candidate_uuid: "c7c8dd9decf94b0eb2498082d63112a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1f3d8ccb5e64924a7327a038fed3756"
I20260927 16:39:13.467639 7709 raft_consensus.cc:3037] T fe2d878f97964cd4bce0ca9c0478ea8b P b1f3d8ccb5e64924a7327a038fed3756 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:13.468719 7709 raft_consensus.cc:2454] T fe2d878f97964cd4bce0ca9c0478ea8b P b1f3d8ccb5e64924a7327a038fed3756 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7c8dd9decf94b0eb2498082d63112a3 in term 1.
I20260927 16:39:13.468710 7572 raft_consensus.cc:2454] T fe2d878f97964cd4bce0ca9c0478ea8b P aa1787383203493a93865001e604596f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7c8dd9decf94b0eb2498082d63112a3 in term 1.
I20260927 16:39:13.468932 7369 leader_election.cc:302] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1f3d8ccb5e64924a7327a038fed3756, c7c8dd9decf94b0eb2498082d63112a3; no voters:
I20260927 16:39:13.469075 7797 raft_consensus.cc:2781] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:39:13.469322 7797 raft_consensus.cc:686] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3 [term 1 LEADER]: Becoming Leader. State: Replica: c7c8dd9decf94b0eb2498082d63112a3, State: Running, Role: LEADER
I20260927 16:39:13.469609 7797 consensus_queue.cc:237] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1787383203493a93865001e604596f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45479 } } peers { permanent_uuid: "b1f3d8ccb5e64924a7327a038fed3756" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33697 } } peers { permanent_uuid: "c7c8dd9decf94b0eb2498082d63112a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42395 } }
I20260927 16:39:13.472483 6510 catalog_manager.cc:5495] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c7c8dd9decf94b0eb2498082d63112a3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c7c8dd9decf94b0eb2498082d63112a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1787383203493a93865001e604596f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1f3d8ccb5e64924a7327a038fed3756" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7c8dd9decf94b0eb2498082d63112a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42395 } health_report { overall_health: HEALTHY } } }
W20260927 16:39:13.484899 7484 tablet.cc:2369] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:39:13.486704 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.482896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44528 (local address 127.0.98.1:42395)
0927 16:39:13.483014 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:13.483018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:13.483030 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:39:13.483124 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:13.483126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:13.483171 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:13.483287 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:39:13.483675 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.484229 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.485239 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.485470 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.485585 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.485918 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:39:13.486303 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:39:13.486441 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:13.486512 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 16:39:13.486565 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:39:13.491870 7709 raft_consensus.cc:1261] T fe2d878f97964cd4bce0ca9c0478ea8b P b1f3d8ccb5e64924a7327a038fed3756 [term 1 FOLLOWER]: Refusing update from remote peer c7c8dd9decf94b0eb2498082d63112a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:39:13.491883 7572 raft_consensus.cc:1261] T fe2d878f97964cd4bce0ca9c0478ea8b P aa1787383203493a93865001e604596f [term 1 FOLLOWER]: Refusing update from remote peer c7c8dd9decf94b0eb2498082d63112a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:39:13.492496 7798 consensus_queue.cc:1035] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1f3d8ccb5e64924a7327a038fed3756" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:39:13.492645 7797 consensus_queue.cc:1035] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa1787383203493a93865001e604596f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:39:13.499739 7801 mvcc.cc:205] Tried to move back new op lower bound from 7333999220696178688 to 7333999220614008832. Current Snapshot: MvccSnapshot[applied={T|T < 7333999220696178688}]
I20260927 16:39:13.503888 7802 mvcc.cc:205] Tried to move back new op lower bound from 7333999220696178688 to 7333999220614008832. Current Snapshot: MvccSnapshot[applied={T|T < 7333999220696178688}]
I20260927 16:39:13.505488 7803 mvcc.cc:205] Tried to move back new op lower bound from 7333999220696178688 to 7333999220614008832. Current Snapshot: MvccSnapshot[applied={T|T < 7333999220696178688 or (T in {7333999220696178688})}]
I20260927 16:39:13.507726 6510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:39:13.507879 6510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:39:13.511981 6510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0dece26d2544ea59c76a1edb6ee1ade: Sending DeleteTablet for 3 replicas of tablet fe2d878f97964cd4bce0ca9c0478ea8b
I20260927 16:39:13.512635 7552 tablet_service.cc:1514] Processing DeleteTablet for tablet fe2d878f97964cd4bce0ca9c0478ea8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:39:13 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55458
I20260927 16:39:13.512867 7689 tablet_service.cc:1514] Processing DeleteTablet for tablet fe2d878f97964cd4bce0ca9c0478ea8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:39:13 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38022
I20260927 16:39:13.513309 7811 tablet_replica.cc:330] T fe2d878f97964cd4bce0ca9c0478ea8b P b1f3d8ccb5e64924a7327a038fed3756: stopping tablet replica
I20260927 16:39:13.513324 7415 tablet_service.cc:1514] Processing DeleteTablet for tablet fe2d878f97964cd4bce0ca9c0478ea8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:39:13 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44518
I20260927 16:39:13.513656 7811 raft_consensus.cc:2229] T fe2d878f97964cd4bce0ca9c0478ea8b P b1f3d8ccb5e64924a7327a038fed3756 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:39:13.513899 7811 raft_consensus.cc:2258] T fe2d878f97964cd4bce0ca9c0478ea8b P b1f3d8ccb5e64924a7327a038fed3756 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:39:13.515851 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 6480
I20260927 16:39:13.517364 7812 tablet_replica.cc:330] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3: stopping tablet replica
I20260927 16:39:13.517671 7812 raft_consensus.cc:2229] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:39:13.518042 7812 raft_consensus.cc:2258] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:39:13.519618 7812 ts_tablet_manager.cc:1905] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:39:13.522029 7811 ts_tablet_manager.cc:1905] T fe2d878f97964cd4bce0ca9c0478ea8b P b1f3d8ccb5e64924a7327a038fed3756: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:39:13.522532 7812 ts_tablet_manager.cc:1918] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:39:13.522624 7812 log.cc:1196] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-0/wal/wals/fe2d878f97964cd4bce0ca9c0478ea8b
I20260927 16:39:13.522949 7812 ts_tablet_manager.cc:1939] T fe2d878f97964cd4bce0ca9c0478ea8b P c7c8dd9decf94b0eb2498082d63112a3: Deleting consensus metadata
W20260927 16:39:13.524333 7370 connection.cc:535] server connection from 127.0.0.1:44518 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 16:39:13.525254 7810 tablet_replica.cc:330] T fe2d878f97964cd4bce0ca9c0478ea8b P aa1787383203493a93865001e604596f: stopping tablet replica
I20260927 16:39:13.525542 7810 raft_consensus.cc:2229] T fe2d878f97964cd4bce0ca9c0478ea8b P aa1787383203493a93865001e604596f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:39:13.525884 7810 raft_consensus.cc:2258] T fe2d878f97964cd4bce0ca9c0478ea8b P aa1787383203493a93865001e604596f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:39:13.527554 7810 ts_tablet_manager.cc:1905] T fe2d878f97964cd4bce0ca9c0478ea8b P aa1787383203493a93865001e604596f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:39:13.528386 7811 ts_tablet_manager.cc:1918] T fe2d878f97964cd4bce0ca9c0478ea8b P b1f3d8ccb5e64924a7327a038fed3756: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:39:13.528486 7811 log.cc:1196] T fe2d878f97964cd4bce0ca9c0478ea8b P b1f3d8ccb5e64924a7327a038fed3756: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/wal/wals/fe2d878f97964cd4bce0ca9c0478ea8b
I20260927 16:39:13.528826 7811 ts_tablet_manager.cc:1939] T fe2d878f97964cd4bce0ca9c0478ea8b P b1f3d8ccb5e64924a7327a038fed3756: Deleting consensus metadata
W20260927 16:39:13.529924 7645 connection.cc:441] server connection from 127.0.0.1:38022 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38022 (request call id 1) could send its response
I20260927 16:39:13.530340 7810 ts_tablet_manager.cc:1918] T fe2d878f97964cd4bce0ca9c0478ea8b P aa1787383203493a93865001e604596f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:39:13.530445 7810 log.cc:1196] T fe2d878f97964cd4bce0ca9c0478ea8b P aa1787383203493a93865001e604596f: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-1/wal/wals/fe2d878f97964cd4bce0ca9c0478ea8b
I20260927 16:39:13.530822 7810 ts_tablet_manager.cc:1939] T fe2d878f97964cd4bce0ca9c0478ea8b P aa1787383203493a93865001e604596f: Deleting consensus metadata
W20260927 16:39:13.532039 7508 connection.cc:441] server connection from 127.0.0.1:55458 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55458 (request call id 1) could send its response
I20260927 16:39:13.533938 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.514640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58632 (local address 127.0.98.61:46569)
0927 16:39:13.514749 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:13.514754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:13.517052 (+ 2298us) server_negotiation.cc:407] Connection header received
0927 16:39:13.517187 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:13.517193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:13.517251 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:13.517326 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:13.519304 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.520098 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.523329 (+ 3231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.523669 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.523752 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.530009 (+ 6257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:13.530013 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:13.530016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:13.530042 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:13.531986 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.532547 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.532553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.532557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.532648 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.533005 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.533008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.533010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.533216 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:13.533337 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:13.533654 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 16:39:13.533793 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 16:39:13.544157 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.532360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43992 (local address 127.0.98.60:46845)
0927 16:39:13.532497 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:13.532502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:13.533822 (+ 1320us) server_negotiation.cc:407] Connection header received
0927 16:39:13.533990 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:13.533993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:13.534043 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:13.534097 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:13.535135 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.536023 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.537357 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.537719 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.537793 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.540208 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:13.540212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:13.540215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:13.540241 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:13.542008 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.542861 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.542864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.542866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.542942 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.543292 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.543296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.543298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.543483 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:13.543602 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:13.543868 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 16:39:13.544005 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 16:39:13.545320 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.533140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58634 (local address 127.0.98.61:46569)
0927 16:39:13.533257 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:13.533263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:13.533666 (+ 403us) server_negotiation.cc:407] Connection header received
0927 16:39:13.533851 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:13.533855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:13.533903 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:13.533964 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:13.534971 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.535918 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.537357 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.537716 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.537794 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.540208 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:13.540212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:13.540215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:13.540241 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:13.542144 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.543993 (+ 1849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.543997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.543999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.544088 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.544396 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.544425 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.544427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.544605 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:13.544754 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:13.544999 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 16:39:13.545153 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:39:13.547081 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:13.514633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43990 (local address 127.0.98.60:46845)
0927 16:39:13.514749 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:13.514754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:13.535374 (+ 20620us) server_negotiation.cc:407] Connection header received
0927 16:39:13.535531 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:13.535534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:13.535593 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:13.535639 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:13.536467 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.537233 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.538388 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:13.539167 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:13.539354 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:13.541932 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:13.541935 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:13.541937 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:13.541995 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:13.545165 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.545861 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.545865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.545867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.545954 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:13.546237 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:13.546241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:13.546243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:13.546435 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:13.546561 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:13.546783 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 16:39:13.546928 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38,"spinlock_wait_cycles":58112}
I20260927 16:39:14.503641 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:14.503141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.3:49069)
0927 16:39:14.503442 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:14.503530 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":121,"threads_started":1}
W20260927 16:39:14.504057 7755 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41911 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
I20260927 16:39:14.505578 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:14.505233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.2:47145)
0927 16:39:14.505443 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:14.505507 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":69,"threads_started":1}
W20260927 16:39:14.505856 7618 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41911 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
I20260927 16:39:14.513729 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:14.512657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.1:36471)
0927 16:39:14.513572 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:14.513645 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":846,"thread_start_us":64,"threads_started":1}
W20260927 16:39:14.513947 7481 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41911 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
I20260927 16:39:15.012298 7824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0dece26d2544ea59c76a1edb6ee1ade)
I20260927 16:39:15.012434 7825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0dece26d2544ea59c76a1edb6ee1ade)
I20260927 16:39:15.012487 7824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: false } }
I20260927 16:39:15.012645 7825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: false } }
I20260927 16:39:15.013505 7824 leader_election.cc:288] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0dece26d2544ea59c76a1edb6ee1ade (127.0.98.62:41911), 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed (127.0.98.61:46569)
I20260927 16:39:15.013526 7825 leader_election.cc:288] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0dece26d2544ea59c76a1edb6ee1ade (127.0.98.62:41911), 7fec907df54a40aeb1da5d546cc4fab8 (127.0.98.60:46845)
I20260927 16:39:15.014041 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:15.013625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.0.1:33580)
0927 16:39:15.013855 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:15.013933 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":58,"threads_started":1}
I20260927 16:39:15.014160 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:15.013628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.0.1:33578)
0927 16:39:15.013885 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:15.013939 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":53,"threads_started":1}
W20260927 16:39:15.014317 7105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
W20260927 16:39:15.014345 7288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
I20260927 16:39:15.015118 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:15.014883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.0.1:33602)
0927 16:39:15.015004 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:15.015047 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260927 16:39:15.015229 7105 leader_election.cc:334] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d0dece26d2544ea59c76a1edb6ee1ade (127.0.98.62:41911): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
I20260927 16:39:15.015480 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:15.015083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.0.1:33614)
0927 16:39:15.015368 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:15.015419 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 16:39:15.015617 7288 leader_election.cc:334] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d0dece26d2544ea59c76a1edb6ee1ade (127.0.98.62:41911): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
I20260927 16:39:15.017707 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:15.013913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44004 (local address 127.0.98.60:46845)
0927 16:39:15.014170 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:15.014174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:15.014501 (+ 327us) server_negotiation.cc:407] Connection header received
0927 16:39:15.014677 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:15.014682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:15.014727 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:15.014813 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:15.015172 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:15.015888 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:15.016850 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:15.017220 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:15.017292 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:15.017363 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:15.017543 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 16:39:15.017589 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":58,"threads_started":1}
I20260927 16:39:15.017719 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:15.014232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58638 (local address 127.0.98.61:46569)
0927 16:39:15.014790 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:15.014794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:15.014809 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:39:15.014854 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:15.014857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:15.014903 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:15.015285 (+ 382us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:15.015381 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:15.016001 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:15.016904 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:15.017240 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:15.017358 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:15.017407 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:15.017538 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 16:39:15.017612 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":490,"thread_start_us":60,"threads_started":1}
I20260927 16:39:15.017730 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:15.014067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46845 (local address 127.0.0.1:44004)
0927 16:39:15.014438 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:15.014452 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:15.014537 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:15.014929 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:15.014932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:15.014952 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:15.015050 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:15.015056 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:15.015992 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:15.015995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:15.016734 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:15.016742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:15.017332 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:15.017335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:15.017410 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:15.017432 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 16:39:15.017521 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 16:39:15.017604 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":1,"thread_start_us":53,"threads_started":1}
I20260927 16:39:15.017716 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:15.013951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46569 (local address 127.0.0.1:58638)
0927 16:39:15.014273 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:15.014288 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:15.014388 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:15.015034 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:15.015037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:15.015054 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:15.015154 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:15.015161 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:15.016093 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:15.016095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:15.016766 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:15.016774 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:15.017357 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:15.017360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:15.017428 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:15.017441 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 16:39:15.017517 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 16:39:15.017587 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":55,"threads_started":1}
I20260927 16:39:15.018141 7309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7fec907df54a40aeb1da5d546cc4fab8" is_pre_election: true
I20260927 16:39:15.018311 7309 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed in term 1.
I20260927 16:39:15.018349 7129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7fec907df54a40aeb1da5d546cc4fab8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" is_pre_election: true
I20260927 16:39:15.018512 7129 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fec907df54a40aeb1da5d546cc4fab8 in term 1.
I20260927 16:39:15.018631 7103 leader_election.cc:302] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed, 7fec907df54a40aeb1da5d546cc4fab8; no voters: d0dece26d2544ea59c76a1edb6ee1ade
I20260927 16:39:15.018841 7825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 16:39:15.018893 7825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 1 FOLLOWER]: Starting leader election (detected failure of leader d0dece26d2544ea59c76a1edb6ee1ade)
I20260927 16:39:15.018920 7825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 1 FOLLOWER]: Advancing to term 2
I20260927 16:39:15.019074 7288 leader_election.cc:302] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed, 7fec907df54a40aeb1da5d546cc4fab8; no voters: d0dece26d2544ea59c76a1edb6ee1ade
I20260927 16:39:15.019291 7824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 16:39:15.019349 7824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d0dece26d2544ea59c76a1edb6ee1ade)
I20260927 16:39:15.019402 7824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [term 1 FOLLOWER]: Advancing to term 2
I20260927 16:39:15.020071 7825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: false } }
I20260927 16:39:15.020380 7824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: false } }
I20260927 16:39:15.020483 7825 leader_election.cc:288] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [CANDIDATE]: Term 2 election: Requested vote from peers d0dece26d2544ea59c76a1edb6ee1ade (127.0.98.62:41911), 7fec907df54a40aeb1da5d546cc4fab8 (127.0.98.60:46845)
I20260927 16:39:15.020726 7824 leader_election.cc:288] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [CANDIDATE]: Term 2 election: Requested vote from peers d0dece26d2544ea59c76a1edb6ee1ade (127.0.98.62:41911), 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed (127.0.98.61:46569)
I20260927 16:39:15.020746 7309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7fec907df54a40aeb1da5d546cc4fab8"
I20260927 16:39:15.020776 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:15.020531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.0.1:33628)
0927 16:39:15.020662 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:15.020709 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 16:39:15.020880 7309 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed in current term 2: Already voted for candidate 7fec907df54a40aeb1da5d546cc4fab8 in this term.
I20260927 16:39:15.021364 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:15.021164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.0.1:33634)
0927 16:39:15.021262 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:15.021309 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 16:39:15.021363 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:15.021171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.0.1:33642)
0927 16:39:15.021262 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:15.021309 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 16:39:15.021550 7105 leader_election.cc:334] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d0dece26d2544ea59c76a1edb6ee1ade (127.0.98.62:41911): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
I20260927 16:39:15.021574 7129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7fec907df54a40aeb1da5d546cc4fab8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed"
I20260927 16:39:15.021641 7105 leader_election.cc:302] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed; no voters: 7fec907df54a40aeb1da5d546cc4fab8, d0dece26d2544ea59c76a1edb6ee1ade
I20260927 16:39:15.021727 7129 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fec907df54a40aeb1da5d546cc4fab8 in current term 2: Already voted for candidate 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed in this term.
I20260927 16:39:15.021884 7825 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 16:39:15.022063 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:15.021822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.0.1:33650)
0927 16:39:15.021923 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:15.022004 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 16:39:15.022158 7288 leader_election.cc:334] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d0dece26d2544ea59c76a1edb6ee1ade (127.0.98.62:41911): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
I20260927 16:39:15.022223 7288 leader_election.cc:302] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fec907df54a40aeb1da5d546cc4fab8; no voters: 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed, d0dece26d2544ea59c76a1edb6ee1ade
I20260927 16:39:15.022343 7824 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 16:39:15.505062 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:15.504620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.3:43541)
0927 16:39:15.504892 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:15.504967 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":81,"threads_started":1}
I20260927 16:39:15.506666 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:15.506326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.2:33171)
0927 16:39:15.506546 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:15.506592 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":78,"threads_started":1}
I20260927 16:39:15.514861 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:15.514480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.1:42865)
0927 16:39:15.514710 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:15.514757 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":88,"threads_started":1}
I20260927 16:39:16.506162 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:16.505742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.3:42289)
0927 16:39:16.505991 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:16.506053 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":89,"threads_started":1}
I20260927 16:39:16.507740 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:16.507332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.2:52899)
0927 16:39:16.507582 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:16.507633 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":60,"threads_started":1}
I20260927 16:39:16.515798 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:16.515480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.1:33111)
0927 16:39:16.515686 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:16.515732 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":52,"threads_started":1}
I20260927 16:39:16.985157 7840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:39:16.985260 7840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: false } }
I20260927 16:39:16.985707 7840 leader_election.cc:288] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0dece26d2544ea59c76a1edb6ee1ade (127.0.98.62:41911), 7fec907df54a40aeb1da5d546cc4fab8 (127.0.98.60:46845)
I20260927 16:39:16.986063 7309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7fec907df54a40aeb1da5d546cc4fab8" is_pre_election: true
I20260927 16:39:16.986110 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:16.985803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.0.1:40662)
0927 16:39:16.985988 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:16.986036 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
I20260927 16:39:16.986177 7309 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed in term 2.
I20260927 16:39:16.986490 7103 leader_election.cc:302] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed, 7fec907df54a40aeb1da5d546cc4fab8; no voters:
I20260927 16:39:16.986694 7840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 16:39:16.986757 7840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:39:16.986825 7840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 2 FOLLOWER]: Advancing to term 3
I20260927 16:39:16.986949 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:16.986701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.0.1:40676)
0927 16:39:16.986830 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:16.986882 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 16:39:16.987059 7105 leader_election.cc:334] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d0dece26d2544ea59c76a1edb6ee1ade (127.0.98.62:41911): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
I20260927 16:39:16.987917 7840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: false } }
I20260927 16:39:16.988257 7840 leader_election.cc:288] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [CANDIDATE]: Term 3 election: Requested vote from peers d0dece26d2544ea59c76a1edb6ee1ade (127.0.98.62:41911), 7fec907df54a40aeb1da5d546cc4fab8 (127.0.98.60:46845)
I20260927 16:39:16.988530 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:16.988356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.0.1:40684)
0927 16:39:16.988434 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:16.988476 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 16:39:16.988549 7309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7fec907df54a40aeb1da5d546cc4fab8"
I20260927 16:39:16.988685 7309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [term 2 FOLLOWER]: Advancing to term 3
I20260927 16:39:16.989063 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:16.988882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.0.1:40700)
0927 16:39:16.988972 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:16.989011 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 16:39:16.989143 7105 leader_election.cc:334] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d0dece26d2544ea59c76a1edb6ee1ade (127.0.98.62:41911): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
I20260927 16:39:16.989477 7309 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed in term 3.
I20260927 16:39:16.989722 7103 leader_election.cc:302] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed, 7fec907df54a40aeb1da5d546cc4fab8; no voters: d0dece26d2544ea59c76a1edb6ee1ade
I20260927 16:39:16.989897 7840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 3 FOLLOWER]: Leader election won for term 3
I20260927 16:39:16.990070 7840 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [term 3 LEADER]: Becoming Leader. State: Replica: 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed, State: Running, Role: LEADER
I20260927 16:39:16.990316 7840 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: false } }
I20260927 16:39:16.991313 7843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed. Latest consensus state: current_term: 3 leader_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: false } } }
I20260927 16:39:16.991423 7843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:16.991690 7845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:39:16.993799 7845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:39:16.994168 7845 catalog_manager.cc:1259] Loaded cluster ID: 7b929cdf13ac47ff88f0adc0945d6ec6
I20260927 16:39:16.994212 7845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:39:16.994733 7845 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:39:16.995087 7845 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed: Loaded TSK: 0
I20260927 16:39:16.995764 7845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:39:17.074513 7119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:39:17.077602 7119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:39:17.080356 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.080105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.0.1:40728)
0927 16:39:17.080224 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:17.080280 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 16:39:17.080456 7309 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [term 3 FOLLOWER]: Refusing update from remote peer 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 16:39:17.080960 7840 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 16:39:17.081051 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.080846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.0.1:40742)
0927 16:39:17.080953 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:17.081002 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 16:39:17.081287 7105 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed -> Peer d0dece26d2544ea59c76a1edb6ee1ade (127.0.98.62:41911): Couldn't send request to peer d0dece26d2544ea59c76a1edb6ee1ade. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 16:39:17.082320 7847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed. Latest consensus state: current_term: 3 leader_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: false } } }
I20260927 16:39:17.082450 7847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:39:17.083398 7843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: false } } }
I20260927 16:39:17.083529 7843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:17.083995 7847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d0dece26d2544ea59c76a1edb6ee1ade" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41911 } } peers { permanent_uuid: "2ad1bd8fcc2b4b0c9c1fdf2dbd875eed" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46569 } attrs { promote: false } } peers { permanent_uuid: "7fec907df54a40aeb1da5d546cc4fab8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46845 } attrs { promote: false } } }
I20260927 16:39:17.084452 7847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fec907df54a40aeb1da5d546cc4fab8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:39:17.092826 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.088811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33697 (local address 127.0.0.1:52456)
0927 16:39:17.089212 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:17.089225 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:17.089311 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:17.089560 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:17.089563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:17.089577 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:17.089672 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:17.089678 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:17.090623 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:17.090629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:17.091735 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:17.091744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:17.092553 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:17.092555 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:17.092622 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:17.092640 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 16:39:17.092697 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 16:39:17.092744 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332}
I20260927 16:39:17.092921 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.088984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52456 (local address 127.0.98.3:33697)
0927 16:39:17.089311 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:17.089317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:17.089331 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:39:17.089416 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:17.089419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:17.089470 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:17.089556 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:17.089768 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:17.090529 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:17.091846 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:17.092428 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:17.092560 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:17.092627 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:17.092729 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 16:39:17.092790 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":101,"threads_started":1}
I20260927 16:39:17.093387 7689 tablet_service.cc:1467] Processing CreateTablet for tablet 3ce9f033cf14494585765d673ef2eda3 (DEFAULT_TABLE table=test-table [id=d1625b2e582048489b670c73d2bdd8c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:39:17.093654 7689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ce9f033cf14494585765d673ef2eda3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:17.095386 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.089011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45479 (local address 127.0.0.1:49786)
0927 16:39:17.089512 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:17.089773 (+ 261us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:17.089842 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:17.090899 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:17.090934 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:17.090953 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:17.091127 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:17.091137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:17.092977 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:17.092981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:17.094213 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:17.094222 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:17.095012 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:17.095017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:17.095106 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:17.095127 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 16:39:17.095187 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 16:39:17.095271 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":99,"thread_start_us":74,"threads_started":1}
I20260927 16:39:17.095718 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.089261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49786 (local address 127.0.98.2:45479)
0927 16:39:17.090678 (+ 1417us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:17.090683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:17.090696 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:39:17.090742 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:17.090745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:17.090789 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:17.090881 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:17.091296 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:17.092854 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:17.094349 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:17.094872 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:17.094997 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:17.095051 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:17.095516 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 16:39:17.095591 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":80,"threads_started":1}
I20260927 16:39:17.096131 7858 tablet_bootstrap.cc:492] T 3ce9f033cf14494585765d673ef2eda3 P b1f3d8ccb5e64924a7327a038fed3756: Bootstrap starting.
I20260927 16:39:17.096325 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.089542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42395 (local address 127.0.0.1:33762)
0927 16:39:17.089894 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:17.089909 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:17.090001 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:17.090237 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:17.090239 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:17.090250 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:17.090318 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:17.090321 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:17.093042 (+ 2721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:17.093046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:17.094213 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:17.094222 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:17.095958 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:17.095963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:17.096049 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:17.096068 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 16:39:17.096139 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 16:39:17.096205 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":13,"thread_start_us":76,"threads_started":1}
I20260927 16:39:17.096371 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.089836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33762 (local address 127.0.98.1:42395)
0927 16:39:17.090050 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:17.090054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:17.090066 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:39:17.090108 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:17.090110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:17.090149 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:17.090225 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:17.090425 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:17.092431 (+ 2006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:17.095105 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:17.095855 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:17.095966 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:17.096025 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:17.096170 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 16:39:17.096254 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":63,"threads_started":1}
I20260927 16:39:17.096375 7552 tablet_service.cc:1467] Processing CreateTablet for tablet 3ce9f033cf14494585765d673ef2eda3 (DEFAULT_TABLE table=test-table [id=d1625b2e582048489b670c73d2bdd8c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:39:17.096603 7552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ce9f033cf14494585765d673ef2eda3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:17.097038 7858 tablet_bootstrap.cc:654] T 3ce9f033cf14494585765d673ef2eda3 P b1f3d8ccb5e64924a7327a038fed3756: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:17.098003 7858 tablet_bootstrap.cc:492] T 3ce9f033cf14494585765d673ef2eda3 P b1f3d8ccb5e64924a7327a038fed3756: No bootstrap required, opened a new log
I20260927 16:39:17.098070 7858 ts_tablet_manager.cc:1397] T 3ce9f033cf14494585765d673ef2eda3 P b1f3d8ccb5e64924a7327a038fed3756: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 16:39:17.098487 7858 raft_consensus.cc:348] T 3ce9f033cf14494585765d673ef2eda3 P b1f3d8ccb5e64924a7327a038fed3756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f3d8ccb5e64924a7327a038fed3756" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33697 } } peers { permanent_uuid: "aa1787383203493a93865001e604596f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45479 } } peers { permanent_uuid: "c7c8dd9decf94b0eb2498082d63112a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42395 } }
I20260927 16:39:17.098659 7860 tablet_bootstrap.cc:492] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f: Bootstrap starting.
I20260927 16:39:17.098706 7858 raft_consensus.cc:374] T 3ce9f033cf14494585765d673ef2eda3 P b1f3d8ccb5e64924a7327a038fed3756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:17.098807 7858 raft_consensus.cc:729] T 3ce9f033cf14494585765d673ef2eda3 P b1f3d8ccb5e64924a7327a038fed3756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1f3d8ccb5e64924a7327a038fed3756, State: Initialized, Role: FOLLOWER
I20260927 16:39:17.099082 7415 tablet_service.cc:1467] Processing CreateTablet for tablet 3ce9f033cf14494585765d673ef2eda3 (DEFAULT_TABLE table=test-table [id=d1625b2e582048489b670c73d2bdd8c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:39:17.099355 7415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ce9f033cf14494585765d673ef2eda3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:17.099560 7860 tablet_bootstrap.cc:654] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:17.100695 7860 tablet_bootstrap.cc:492] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f: No bootstrap required, opened a new log
I20260927 16:39:17.100770 7860 ts_tablet_manager.cc:1397] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 16:39:17.101150 7860 raft_consensus.cc:348] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f3d8ccb5e64924a7327a038fed3756" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33697 } } peers { permanent_uuid: "aa1787383203493a93865001e604596f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45479 } } peers { permanent_uuid: "c7c8dd9decf94b0eb2498082d63112a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42395 } }
I20260927 16:39:17.101254 7860 raft_consensus.cc:374] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:17.101293 7860 raft_consensus.cc:729] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa1787383203493a93865001e604596f, State: Initialized, Role: FOLLOWER
I20260927 16:39:17.101420 7860 consensus_queue.cc:260] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f3d8ccb5e64924a7327a038fed3756" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33697 } } peers { permanent_uuid: "aa1787383203493a93865001e604596f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45479 } } peers { permanent_uuid: "c7c8dd9decf94b0eb2498082d63112a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42395 } }
I20260927 16:39:17.101763 7860 ts_tablet_manager.cc:1428] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:39:17.101815 7863 tablet_bootstrap.cc:492] T 3ce9f033cf14494585765d673ef2eda3 P c7c8dd9decf94b0eb2498082d63112a3: Bootstrap starting.
I20260927 16:39:17.102192 7858 consensus_queue.cc:260] T 3ce9f033cf14494585765d673ef2eda3 P b1f3d8ccb5e64924a7327a038fed3756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f3d8ccb5e64924a7327a038fed3756" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33697 } } peers { permanent_uuid: "aa1787383203493a93865001e604596f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45479 } } peers { permanent_uuid: "c7c8dd9decf94b0eb2498082d63112a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42395 } }
I20260927 16:39:17.102581 7858 ts_tablet_manager.cc:1428] T 3ce9f033cf14494585765d673ef2eda3 P b1f3d8ccb5e64924a7327a038fed3756: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260927 16:39:17.102653 7863 tablet_bootstrap.cc:654] T 3ce9f033cf14494585765d673ef2eda3 P c7c8dd9decf94b0eb2498082d63112a3: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:17.104166 7863 tablet_bootstrap.cc:492] T 3ce9f033cf14494585765d673ef2eda3 P c7c8dd9decf94b0eb2498082d63112a3: No bootstrap required, opened a new log
I20260927 16:39:17.104251 7863 ts_tablet_manager.cc:1397] T 3ce9f033cf14494585765d673ef2eda3 P c7c8dd9decf94b0eb2498082d63112a3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:17.104350 7620 heartbeater.cc:503] Master 127.0.98.61:46569 was elected leader, sending a full tablet report...
I20260927 16:39:17.104497 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.102972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.3:60419)
0927 16:39:17.104336 (+ 1364us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:17.104390 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1307,"thread_start_us":76,"threads_started":1}
I20260927 16:39:17.104632 7863 raft_consensus.cc:348] T 3ce9f033cf14494585765d673ef2eda3 P c7c8dd9decf94b0eb2498082d63112a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f3d8ccb5e64924a7327a038fed3756" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33697 } } peers { permanent_uuid: "aa1787383203493a93865001e604596f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45479 } } peers { permanent_uuid: "c7c8dd9decf94b0eb2498082d63112a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42395 } }
I20260927 16:39:17.104729 7863 raft_consensus.cc:374] T 3ce9f033cf14494585765d673ef2eda3 P c7c8dd9decf94b0eb2498082d63112a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:17.104764 7863 raft_consensus.cc:729] T 3ce9f033cf14494585765d673ef2eda3 P c7c8dd9decf94b0eb2498082d63112a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7c8dd9decf94b0eb2498082d63112a3, State: Initialized, Role: FOLLOWER
I20260927 16:39:17.104876 7863 consensus_queue.cc:260] T 3ce9f033cf14494585765d673ef2eda3 P c7c8dd9decf94b0eb2498082d63112a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f3d8ccb5e64924a7327a038fed3756" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33697 } } peers { permanent_uuid: "aa1787383203493a93865001e604596f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45479 } } peers { permanent_uuid: "c7c8dd9decf94b0eb2498082d63112a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42395 } }
I20260927 16:39:17.105046 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.101992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.2:60917)
0927 16:39:17.104907 (+ 2915us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:17.104958 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2857,"thread_start_us":80,"threads_started":1}
I20260927 16:39:17.105170 7863 ts_tablet_manager.cc:1428] T 3ce9f033cf14494585765d673ef2eda3 P c7c8dd9decf94b0eb2498082d63112a3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:39:17.105892 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.105401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.2:48365)
0927 16:39:17.105765 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:17.105825 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":74,"threads_started":1}
I20260927 16:39:17.105993 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.105735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.3:44703)
0927 16:39:17.105884 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:17.105934 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 16:39:17.106315 7757 heartbeater.cc:503] Master 127.0.98.61:46569 was elected leader, sending a full tablet report...
I20260927 16:39:17.106665 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.106270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.1:50207)
0927 16:39:17.106512 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:17.106566 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":86,"threads_started":1}
I20260927 16:39:17.107729 7483 heartbeater.cc:503] Master 127.0.98.61:46569 was elected leader, sending a full tablet report...
I20260927 16:39:17.107802 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.107564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.1:40329)
0927 16:39:17.107700 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:17.107745 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 16:39:17.122401 7862 raft_consensus.cc:482] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:39:17.122504 7862 raft_consensus.cc:504] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f3d8ccb5e64924a7327a038fed3756" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33697 } } peers { permanent_uuid: "aa1787383203493a93865001e604596f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45479 } } peers { permanent_uuid: "c7c8dd9decf94b0eb2498082d63112a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42395 } }
I20260927 16:39:17.123469 7862 leader_election.cc:288] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1f3d8ccb5e64924a7327a038fed3756 (127.0.98.3:33697), c7c8dd9decf94b0eb2498082d63112a3 (127.0.98.1:42395)
I20260927 16:39:17.127260 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.123649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33697 (local address 127.0.98.2:52995)
0927 16:39:17.123743 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:17.123757 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:17.123859 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:17.124166 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:17.124168 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:17.124180 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:17.124277 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:17.124283 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:17.125173 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:17.125175 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:17.126161 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:17.126174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:17.126842 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:17.126846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:17.126931 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:17.126958 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 16:39:17.127032 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 16:39:17.127100 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 16:39:17.127265 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.123788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52995 (local address 127.0.98.3:33697)
0927 16:39:17.123910 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:17.123916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:17.123930 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:39:17.123975 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:17.123978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:17.124020 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:17.124091 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:17.124380 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:17.125074 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:17.126330 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:17.126727 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:17.126851 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:17.126923 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:17.127072 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 16:39:17.127133 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 16:39:17.127534 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.123887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42395 (local address 127.0.98.2:55827)
0927 16:39:17.124009 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:17.124020 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:17.124088 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:17.124399 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:17.124401 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:17.124418 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:17.124496 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:17.124500 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:17.125328 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:17.125334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:17.126189 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:17.126199 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:17.127084 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:17.127087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:17.127164 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:17.127185 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 16:39:17.127278 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 16:39:17.127435 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 16:39:17.127543 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.124112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55827 (local address 127.0.98.1:42395)
0927 16:39:17.124207 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:17.124211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:17.124221 (+ 10us) server_negotiation.cc:407] Connection header received
0927 16:39:17.124262 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:17.124264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:17.124294 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:17.124382 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:17.124602 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:17.125228 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:17.126378 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:17.126944 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:17.127051 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:17.127108 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:17.127315 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 16:39:17.127426 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 16:39:17.127741 7709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce9f033cf14494585765d673ef2eda3" candidate_uuid: "aa1787383203493a93865001e604596f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1f3d8ccb5e64924a7327a038fed3756" is_pre_election: true
I20260927 16:39:17.127869 7709 raft_consensus.cc:2454] T 3ce9f033cf14494585765d673ef2eda3 P b1f3d8ccb5e64924a7327a038fed3756 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa1787383203493a93865001e604596f in term 0.
I20260927 16:39:17.128023 7435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce9f033cf14494585765d673ef2eda3" candidate_uuid: "aa1787383203493a93865001e604596f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7c8dd9decf94b0eb2498082d63112a3" is_pre_election: true
I20260927 16:39:17.128237 7435 raft_consensus.cc:2454] T 3ce9f033cf14494585765d673ef2eda3 P c7c8dd9decf94b0eb2498082d63112a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa1787383203493a93865001e604596f in term 0.
I20260927 16:39:17.128242 7506 leader_election.cc:302] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa1787383203493a93865001e604596f, b1f3d8ccb5e64924a7327a038fed3756; no voters:
I20260927 16:39:17.128468 7862 raft_consensus.cc:2781] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:39:17.128531 7862 raft_consensus.cc:482] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:39:17.128592 7862 raft_consensus.cc:3037] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:17.129568 7862 raft_consensus.cc:504] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f3d8ccb5e64924a7327a038fed3756" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33697 } } peers { permanent_uuid: "aa1787383203493a93865001e604596f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45479 } } peers { permanent_uuid: "c7c8dd9decf94b0eb2498082d63112a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42395 } }
I20260927 16:39:17.129885 7862 leader_election.cc:288] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f [CANDIDATE]: Term 1 election: Requested vote from peers b1f3d8ccb5e64924a7327a038fed3756 (127.0.98.3:33697), c7c8dd9decf94b0eb2498082d63112a3 (127.0.98.1:42395)
I20260927 16:39:17.130173 7709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce9f033cf14494585765d673ef2eda3" candidate_uuid: "aa1787383203493a93865001e604596f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1f3d8ccb5e64924a7327a038fed3756"
I20260927 16:39:17.130249 7435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce9f033cf14494585765d673ef2eda3" candidate_uuid: "aa1787383203493a93865001e604596f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7c8dd9decf94b0eb2498082d63112a3"
I20260927 16:39:17.130323 7709 raft_consensus.cc:3037] T 3ce9f033cf14494585765d673ef2eda3 P b1f3d8ccb5e64924a7327a038fed3756 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:17.130381 7435 raft_consensus.cc:3037] T 3ce9f033cf14494585765d673ef2eda3 P c7c8dd9decf94b0eb2498082d63112a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:17.131165 7709 raft_consensus.cc:2454] T 3ce9f033cf14494585765d673ef2eda3 P b1f3d8ccb5e64924a7327a038fed3756 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa1787383203493a93865001e604596f in term 1.
I20260927 16:39:17.131359 7435 raft_consensus.cc:2454] T 3ce9f033cf14494585765d673ef2eda3 P c7c8dd9decf94b0eb2498082d63112a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa1787383203493a93865001e604596f in term 1.
I20260927 16:39:17.131528 7506 leader_election.cc:302] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa1787383203493a93865001e604596f, b1f3d8ccb5e64924a7327a038fed3756; no voters:
I20260927 16:39:17.131721 7862 raft_consensus.cc:2781] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f [term 1 FOLLOWER]: Leader election won for term 1
W20260927 16:39:17.131925 7758 tablet.cc:2369] T 3ce9f033cf14494585765d673ef2eda3 P b1f3d8ccb5e64924a7327a038fed3756: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:39:17.131929 7862 raft_consensus.cc:686] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f [term 1 LEADER]: Becoming Leader. State: Replica: aa1787383203493a93865001e604596f, State: Running, Role: LEADER
I20260927 16:39:17.132580 7862 consensus_queue.cc:237] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f3d8ccb5e64924a7327a038fed3756" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33697 } } peers { permanent_uuid: "aa1787383203493a93865001e604596f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45479 } } peers { permanent_uuid: "c7c8dd9decf94b0eb2498082d63112a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42395 } }
I20260927 16:39:17.133561 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.132253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.2:54659)
0927 16:39:17.133476 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:17.133516 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1179}
I20260927 16:39:17.134788 7119 catalog_manager.cc:5495] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f reported cstate change: term changed from 0 to 1, leader changed from <none> to aa1787383203493a93865001e604596f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "aa1787383203493a93865001e604596f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f3d8ccb5e64924a7327a038fed3756" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa1787383203493a93865001e604596f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7c8dd9decf94b0eb2498082d63112a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42395 } health_report { overall_health: UNKNOWN } } }
I20260927 16:39:17.154223 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.150606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49796 (local address 127.0.98.2:45479)
0927 16:39:17.150741 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:17.150746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:17.150758 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:39:17.150898 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:17.150900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:17.150952 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:17.151060 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:39:17.151343 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:17.151838 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:17.152737 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:17.153006 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:17.153105 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:17.153365 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:39:17.153840 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:39:17.153957 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:17.154024 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 16:39:17.154070 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:39:17.156898 7709 raft_consensus.cc:1261] T 3ce9f033cf14494585765d673ef2eda3 P b1f3d8ccb5e64924a7327a038fed3756 [term 1 FOLLOWER]: Refusing update from remote peer aa1787383203493a93865001e604596f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:39:17.157070 7435 raft_consensus.cc:1261] T 3ce9f033cf14494585765d673ef2eda3 P c7c8dd9decf94b0eb2498082d63112a3 [term 1 FOLLOWER]: Refusing update from remote peer aa1787383203493a93865001e604596f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:39:17.157388 7862 consensus_queue.cc:1035] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1f3d8ccb5e64924a7327a038fed3756" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:39:17.157565 7871 consensus_queue.cc:1035] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7c8dd9decf94b0eb2498082d63112a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:39:17.159130 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.158841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.3:44837)
0927 16:39:17.158982 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:17.159052 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 16:39:17.160156 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.159828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.1:35037)
0927 16:39:17.160032 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:17.160086 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260927 16:39:17.162922 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.162701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.2:51621)
0927 16:39:17.162787 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:17.162854 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 16:39:17.164229 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.163960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.3:37269)
0927 16:39:17.164110 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:17.164161 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 16:39:17.166471 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.166118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.1:53795)
0927 16:39:17.166360 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:17.166412 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260927 16:39:17.168483 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.165579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41911 (local address 127.0.98.2:54373)
0927 16:39:17.168378 (+ 2799us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:17.168426 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2741}
I20260927 16:39:17.170539 7118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:39:17.170688 7118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:39:17.174142 7118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ad1bd8fcc2b4b0c9c1fdf2dbd875eed: Sending DeleteTablet for 3 replicas of tablet 3ce9f033cf14494585765d673ef2eda3
I20260927 16:39:17.174741 7689 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ce9f033cf14494585765d673ef2eda3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:39:17 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:52456
I20260927 16:39:17.174793 7415 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ce9f033cf14494585765d673ef2eda3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:39:17 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33762
I20260927 16:39:17.175365 7884 tablet_replica.cc:330] T 3ce9f033cf14494585765d673ef2eda3 P b1f3d8ccb5e64924a7327a038fed3756: stopping tablet replica
I20260927 16:39:17.175505 7884 raft_consensus.cc:2229] T 3ce9f033cf14494585765d673ef2eda3 P b1f3d8ccb5e64924a7327a038fed3756 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:39:17.175495 7552 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ce9f033cf14494585765d673ef2eda3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:39:17 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:49786
I20260927 16:39:17.175613 7884 raft_consensus.cc:2258] T 3ce9f033cf14494585765d673ef2eda3 P b1f3d8ccb5e64924a7327a038fed3756 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:39:17.175762 7886 tablet_replica.cc:330] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f: stopping tablet replica
I20260927 16:39:17.175812 7885 tablet_replica.cc:330] T 3ce9f033cf14494585765d673ef2eda3 P c7c8dd9decf94b0eb2498082d63112a3: stopping tablet replica
I20260927 16:39:17.175879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 7263
I20260927 16:39:17.176530 7884 ts_tablet_manager.cc:1905] T 3ce9f033cf14494585765d673ef2eda3 P b1f3d8ccb5e64924a7327a038fed3756: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:39:17.179570 7884 ts_tablet_manager.cc:1918] T 3ce9f033cf14494585765d673ef2eda3 P b1f3d8ccb5e64924a7327a038fed3756: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:39:17.179682 7886 raft_consensus.cc:2229] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:39:17.180045 7886 raft_consensus.cc:2258] T 3ce9f033cf14494585765d673ef2eda3 P aa1787383203493a93865001e604596f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:39:17.180248 7884 log.cc:1196] T 3ce9f033cf14494585765d673ef2eda3 P b1f3d8ccb5e64924a7327a038fed3756: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790527121515808-392-0/minicluster-data/ts-2/wal/wals/3ce9f033cf14494585765d673ef2eda3
I20260927 16:39:17.180631 7884 ts_tablet_manager.cc:1939] T 3ce9f033cf14494585765d673ef2eda3 P b1f3d8ccb5e64924a7327a038fed3756: Deleting consensus metadata
I20260927 16:39:17.185086 7104 catalog_manager.cc:4841] TS b1f3d8ccb5e64924a7327a038fed3756 (127.0.98.3:33697): tablet 3ce9f033cf14494585765d673ef2eda3 (table test-table [id=d1625b2e582048489b670c73d2bdd8c2]) successfully deleted
I20260927 16:39:17.185437 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 7489
I20260927 16:39:17.186404 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.185603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42395 (local address 127.0.0.1:33768)
0927 16:39:17.186305 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:17.186348 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":633}
W20260927 16:39:17.186550 7104 catalog_manager.cc:4584] TS c7c8dd9decf94b0eb2498082d63112a3 (127.0.98.1:42395): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3ce9f033cf14494585765d673ef2eda3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42395: connect: Connection refused (error 111)
I20260927 16:39:17.192245 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 7625
I20260927 16:39:17.192653 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:17.192388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45479 (local address 127.0.0.1:49810)
0927 16:39:17.192536 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:17.192583 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 16:39:17.197587 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 6976
I20260927 16:39:17.203148 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 7083
2026-09-27T16:39:17Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6992 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:39:17 dist-test-slave-z40b krb5kdc[7889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:39:17 dist-test-slave-z40b krb5kdc[7889](info): set up 2 sockets
Sep 27 16:39:17 dist-test-slave-z40b krb5kdc[7889](info): commencing operation
krb5kdc: starting...
W20260927 16:39:18.275763 392 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:39:18 dist-test-slave-z40b krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:39:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:39:18Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 16:39:18.388454 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38577
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:18.490993 7902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:18.491246 7902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:18.491285 7902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:18.495126 7902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:39:18.495178 7902 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 16:39:18.495194 7902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:18.495227 7902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:18.495242 7902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:39:18.495257 7902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:39:18.500120 7902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38577
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:18.501093 7902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:18.502055 7902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:18.509984 7910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:18.509989 7908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:18.510141 7902 server_base.cc:1034] running on GCE node
W20260927 16:39:18.510071 7907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:18.510797 7902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:18.511933 7902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:18.513113 7902 hybrid_clock.cc:648] HybridClock initialized: now 1790527158513088 us; error 37 us; skew 500 ppm
Sep 27 16:39:18 dist-test-slave-z40b krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:18.515683 7902 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 16:39:18.516507 7902 webserver.cc:460] Webserver started at http://127.0.98.62:38711/ using document root <none> and password file <none>
I20260927 16:39:18.517061 7902 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:18.517148 7902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:18.517371 7902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:18.519327 7902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "0ba0e51bba58480d8956cbc6689b1379"
format_stamp: "Formatted at 2026-09-27 16:39:18 on dist-test-slave-z40b"
server_key: "f04d78a017c567ff563aa76950553bdc"
server_key_iv: "b9c0156112783914636c1b816546d1ba"
server_key_version: "encryptionkey@0"
I20260927 16:39:18.519825 7902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "0ba0e51bba58480d8956cbc6689b1379"
format_stamp: "Formatted at 2026-09-27 16:39:18 on dist-test-slave-z40b"
server_key: "f04d78a017c567ff563aa76950553bdc"
server_key_iv: "b9c0156112783914636c1b816546d1ba"
server_key_version: "encryptionkey@0"
I20260927 16:39:18.523164 7902 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:39:18.525300 7917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:18.526427 7902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:18.526556 7902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "0ba0e51bba58480d8956cbc6689b1379"
format_stamp: "Formatted at 2026-09-27 16:39:18 on dist-test-slave-z40b"
server_key: "f04d78a017c567ff563aa76950553bdc"
server_key_iv: "b9c0156112783914636c1b816546d1ba"
server_key_version: "encryptionkey@0"
I20260927 16:39:18.526664 7902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:18.543654 7902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:18.546599 7902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:18.546783 7902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:18.554113 7902 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38577
I20260927 16:39:18.554119 7979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38577 every 8 connection(s)
I20260927 16:39:18.554943 7902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:39:18.557403 7980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:18.562229 7980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba0e51bba58480d8956cbc6689b1379: Bootstrap starting.
I20260927 16:39:18.563395 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 7902
I20260927 16:39:18.563521 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:39:18.563786 392 external_mini_cluster.cc:1428] Setting key da67528a3def4dd57c108d437a7f11f6
I20260927 16:39:18.564713 7980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ba0e51bba58480d8956cbc6689b1379: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:18.565479 7980 log.cc:824] T 00000000000000000000000000000000 P 0ba0e51bba58480d8956cbc6689b1379: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:18.567536 7980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba0e51bba58480d8956cbc6689b1379: No bootstrap required, opened a new log
I20260927 16:39:18.571559 7980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ba0e51bba58480d8956cbc6689b1379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba0e51bba58480d8956cbc6689b1379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38577 } }
I20260927 16:39:18.571832 7980 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ba0e51bba58480d8956cbc6689b1379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:18.571918 7980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ba0e51bba58480d8956cbc6689b1379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ba0e51bba58480d8956cbc6689b1379, State: Initialized, Role: FOLLOWER
I20260927 16:39:18.572295 7980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ba0e51bba58480d8956cbc6689b1379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba0e51bba58480d8956cbc6689b1379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38577 } }
Sep 27 16:39:18 dist-test-slave-z40b krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 16:39:18.572438 7980 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ba0e51bba58480d8956cbc6689b1379 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:39:18.572511 7980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ba0e51bba58480d8956cbc6689b1379 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:39:18.572603 7980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ba0e51bba58480d8956cbc6689b1379 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:18.573473 7980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ba0e51bba58480d8956cbc6689b1379 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba0e51bba58480d8956cbc6689b1379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38577 } }
I20260927 16:39:18.573782 7980 leader_election.cc:302] T 00000000000000000000000000000000 P 0ba0e51bba58480d8956cbc6689b1379 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ba0e51bba58480d8956cbc6689b1379; no voters:
I20260927 16:39:18.574131 7980 leader_election.cc:288] T 00000000000000000000000000000000 P 0ba0e51bba58480d8956cbc6689b1379 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:39:18.574245 7985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ba0e51bba58480d8956cbc6689b1379 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:39:18.574446 7985 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ba0e51bba58480d8956cbc6689b1379 [term 1 LEADER]: Becoming Leader. State: Replica: 0ba0e51bba58480d8956cbc6689b1379, State: Running, Role: LEADER
I20260927 16:39:18.574753 7985 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ba0e51bba58480d8956cbc6689b1379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba0e51bba58480d8956cbc6689b1379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38577 } }
I20260927 16:39:18.575367 7980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ba0e51bba58480d8956cbc6689b1379 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:39:18.576628 7987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba0e51bba58480d8956cbc6689b1379 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ba0e51bba58480d8956cbc6689b1379. Latest consensus state: current_term: 1 leader_uuid: "0ba0e51bba58480d8956cbc6689b1379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba0e51bba58480d8956cbc6689b1379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38577 } } }
I20260927 16:39:18.576653 7986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba0e51bba58480d8956cbc6689b1379 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ba0e51bba58480d8956cbc6689b1379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba0e51bba58480d8956cbc6689b1379" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38577 } } }
I20260927 16:39:18.576836 7986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba0e51bba58480d8956cbc6689b1379 [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:18.577224 7995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:39:18.576826 7987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba0e51bba58480d8956cbc6689b1379 [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:18.578640 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:18.564774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34344 (local address 127.0.98.62:38577)
0927 16:39:18.565192 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:18.565206 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:18.565237 (+ 31us) server_negotiation.cc:407] Connection header received
0927 16:39:18.565946 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:18.565969 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:18.566326 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:18.566578 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:18.567467 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:18.568402 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:18.569653 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:18.570104 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:18.570211 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:18.572990 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:18.572994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:18.573006 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:18.573029 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:18.574992 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:18.575512 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:18.575517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:18.575522 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:18.575643 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:18.576017 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:18.576020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:18.576022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:18.576532 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:18.577247 (+ 715us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:18.577408 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 16:39:18.577751 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":127,"threads_started":1}
I20260927 16:39:18.580139 7995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:39:18.584611 7995 catalog_manager.cc:1347] Generated new cluster ID: 8cc99f0a1b3b4bd0a84b18060741998b
I20260927 16:39:18.584668 7995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:39:18.617508 7995 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:39:18.618310 7995 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:39:18.626019 7995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ba0e51bba58480d8956cbc6689b1379: Generated new TSK 0
I20260927 16:39:18.626593 7995 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 16:39:18.658018 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38577
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:18.776806 8006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:18.777014 8006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:18.777087 8006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:18.780831 8006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:18.780874 8006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:18.780910 8006 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 16:39:18.780946 8006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:39:18.785856 8006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:18.786651 8006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:18.787505 8006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:18.795936 8011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:18.796021 8012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:18.796190 8006 server_base.cc:1034] running on GCE node
W20260927 16:39:18.796011 8014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:18.796566 8006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:18.797148 8006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:18.798323 8006 hybrid_clock.cc:648] HybridClock initialized: now 1790527158798311 us; error 23 us; skew 500 ppm
Sep 27 16:39:18 dist-test-slave-z40b krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:18.800966 8006 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 16:39:18.801672 8006 webserver.cc:460] Webserver started at http://127.0.98.1:35497/ using document root <none> and password file <none>
I20260927 16:39:18.802146 8006 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:18.802201 8006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:18.802346 8006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:18.804232 8006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-0/data/instance:
uuid: "ad11280495cd4070af56c658de93b7ec"
format_stamp: "Formatted at 2026-09-27 16:39:18 on dist-test-slave-z40b"
server_key: "8151b34b2cfa66216f9df88d14c1c60e"
server_key_iv: "aa4d068f42614284781543635176b735"
server_key_version: "encryptionkey@0"
I20260927 16:39:18.804704 8006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ad11280495cd4070af56c658de93b7ec"
format_stamp: "Formatted at 2026-09-27 16:39:18 on dist-test-slave-z40b"
server_key: "8151b34b2cfa66216f9df88d14c1c60e"
server_key_iv: "aa4d068f42614284781543635176b735"
server_key_version: "encryptionkey@0"
I20260927 16:39:18.807982 8006 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:39:18.810038 8021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:18.811168 8006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:18.811326 8006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "ad11280495cd4070af56c658de93b7ec"
format_stamp: "Formatted at 2026-09-27 16:39:18 on dist-test-slave-z40b"
server_key: "8151b34b2cfa66216f9df88d14c1c60e"
server_key_iv: "aa4d068f42614284781543635176b735"
server_key_version: "encryptionkey@0"
I20260927 16:39:18.811429 8006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:18.832866 8006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:18.836190 8006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:18.836393 8006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:18.837742 8006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:18.837836 8006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:18.837893 8006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:18.837944 8006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 16:39:18 dist-test-slave-z40b krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 16:39:18.848503 8006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36701
I20260927 16:39:18.848574 8136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36701 every 8 connection(s)
I20260927 16:39:18.849567 8006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:39:18.852692 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:18.838556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37735 (local address 127.0.98.62:38577)
0927 16:39:18.838708 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:18.838713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:18.839613 (+ 900us) server_negotiation.cc:407] Connection header received
0927 16:39:18.840773 (+ 1160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:18.840779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:18.840844 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:18.840962 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:18.842456 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:18.843330 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:18.845257 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:18.845619 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:18.845702 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:18.848668 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:18.848672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:18.848675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:18.848718 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:18.850187 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:18.850821 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:18.850825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:18.850827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:18.850942 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:18.851414 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:18.851418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:18.851421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:18.851635 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:18.851724 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:18.852391 (+ 667us) server_negotiation.cc:299] Negotiation successful
0927 16:39:18.852533 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 16:39:18.853559 8030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:18.838978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38577 (local address 127.0.98.1:37735)
0927 16:39:18.839474 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:18.839522 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:18.840552 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:18.841135 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:18.841143 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:18.841621 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:18.842296 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:18.842340 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:18.843488 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:18.843494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:18.845115 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:18.845129 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:18.845781 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:18.845786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:18.846018 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:18.846658 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:18.846683 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:18.848444 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:18.850307 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:18.850312 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:18.850324 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:18.850701 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:18.851044 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:18.851047 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:18.851049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:18.851278 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:18.851808 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:18.851813 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:18.852105 (+ 292us) client_negotiation.cc:769] Sending connection context
0927 16:39:18.852283 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 16:39:18.852639 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":136,"threads_started":1}
I20260927 16:39:18.854712 8137 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38577
I20260927 16:39:18.854863 8137 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:18.855427 8137 heartbeater.cc:511] Master 127.0.98.62:38577 requested a full tablet report, sending...
I20260927 16:39:18.856978 7933 ts_manager.cc:194] Registered new tserver with Master: ad11280495cd4070af56c658de93b7ec (127.0.98.1:36701)
I20260927 16:39:18.858047 7933 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37735
I20260927 16:39:18.859534 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 8006
I20260927 16:39:18.859668 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:39:18.859980 392 external_mini_cluster.cc:1428] Setting key ab7b996106d04c0b45b7d2a73eebec24
I20260927 16:39:18.863312 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:18.856681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43061 (local address 127.0.98.62:38577)
0927 16:39:18.856805 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:18.856810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:18.856970 (+ 160us) server_negotiation.cc:407] Connection header received
0927 16:39:18.857076 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:18.857080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:18.857119 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:18.857195 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:18.857569 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:18.858056 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:18.858932 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:18.859156 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:18.859279 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:18.860346 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:18.860349 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:18.860351 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:18.860377 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:18.861736 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:18.862143 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:18.862148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:18.862150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:18.862221 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:18.862501 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:18.862504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:18.862506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:18.862651 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:18.862773 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:18.863031 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 16:39:18.863126 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":47}
I20260927 16:39:18.863320 8143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:18.856624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38577 (local address 127.0.98.1:43061)
0927 16:39:18.856897 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:18.856908 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:18.857007 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:18.857207 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:18.857210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:18.857385 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:18.857466 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:18.857470 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:18.858171 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:18.858174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:18.858802 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:18.858811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:18.859278 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:18.859281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:18.859369 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:18.859831 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:18.859844 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:18.860222 (+ 378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:18.861857 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:18.861861 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:18.861863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:18.862046 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:18.862319 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:18.862323 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:18.862328 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:18.862409 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:18.862768 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:18.862771 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:18.862868 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 16:39:18.862983 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 16:39:18.863101 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 16:39:18.885950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38577
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:18.987787 8146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:18.988005 8146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:18.988045 8146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:18.991896 8146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:18.991936 8146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:18.991971 8146 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 16:39:18.992002 8146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:39:18.996649 8146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:18.997419 8146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:18.998256 8146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:19.006675 8154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:19.006745 8146 server_base.cc:1034] running on GCE node
W20260927 16:39:19.006717 8151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:19.006887 8152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:19.007305 8146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:19.007860 8146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:19.009068 8146 hybrid_clock.cc:648] HybridClock initialized: now 1790527159009051 us; error 31 us; skew 500 ppm
Sep 27 16:39:19 dist-test-slave-z40b krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527159, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:19.011785 8146 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 16:39:19.012502 8146 webserver.cc:460] Webserver started at http://127.0.98.2:41773/ using document root <none> and password file <none>
I20260927 16:39:19.012969 8146 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:19.013022 8146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:19.013168 8146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:19.015079 8146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-1/data/instance:
uuid: "34362613bae544069a507d0f45be0de5"
format_stamp: "Formatted at 2026-09-27 16:39:19 on dist-test-slave-z40b"
server_key: "fc431bc473bd181e65af94c6be9d6b6a"
server_key_iv: "1fd34ac13ede17e656a61890937f5eeb"
server_key_version: "encryptionkey@0"
I20260927 16:39:19.015627 8146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-1/wal/instance:
uuid: "34362613bae544069a507d0f45be0de5"
format_stamp: "Formatted at 2026-09-27 16:39:19 on dist-test-slave-z40b"
server_key: "fc431bc473bd181e65af94c6be9d6b6a"
server_key_iv: "1fd34ac13ede17e656a61890937f5eeb"
server_key_version: "encryptionkey@0"
I20260927 16:39:19.018872 8146 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:39:19.021093 8161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:19.022224 8146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:19.022313 8146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "34362613bae544069a507d0f45be0de5"
format_stamp: "Formatted at 2026-09-27 16:39:19 on dist-test-slave-z40b"
server_key: "fc431bc473bd181e65af94c6be9d6b6a"
server_key_iv: "1fd34ac13ede17e656a61890937f5eeb"
server_key_version: "encryptionkey@0"
I20260927 16:39:19.022389 8146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:19.032796 8146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:19.035643 8146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:19.035771 8146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:19.036962 8146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:19.037009 8146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:19.037043 8146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:19.037094 8146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 16:39:19 dist-test-slave-z40b krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527159, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 16:39:19.047415 8146 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40193
I20260927 16:39:19.047487 8276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40193 every 8 connection(s)
I20260927 16:39:19.048305 8146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:39:19.050266 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:19.037544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35913 (local address 127.0.98.62:38577)
0927 16:39:19.037681 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:19.037687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:19.038427 (+ 740us) server_negotiation.cc:407] Connection header received
0927 16:39:19.039386 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:19.039390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:19.039441 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:19.039499 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:19.041243 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.042102 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.043433 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.043665 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.043776 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:19.046690 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:19.046693 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:19.046695 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:19.046717 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:19.048284 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:19.048843 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:19.048850 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:19.048853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:19.048966 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:19.049327 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:19.049330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:19.049331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:19.049488 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:19.049564 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:19.049958 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 16:39:19.050068 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:39:19.051007 8170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:19.037876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38577 (local address 127.0.98.2:35913)
0927 16:39:19.038268 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:19.038312 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:19.039129 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:19.039932 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:19.039940 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:19.040402 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:19.041053 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:19.041073 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.042212 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.042215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:19.043313 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:19.043322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.043819 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.043823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:19.043995 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:19.044510 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:19.044529 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:19.046518 (+ 1989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:19.048394 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:19.048399 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:19.048411 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:19.048727 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:19.049066 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:19.049069 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:19.049071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:19.049235 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:19.049575 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:19.049579 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:19.049768 (+ 189us) client_negotiation.cc:769] Sending connection context
0927 16:39:19.049912 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 16:39:19.050216 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":117,"threads_started":1}
I20260927 16:39:19.051434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 8146
I20260927 16:39:19.051577 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:39:19.051903 392 external_mini_cluster.cc:1428] Setting key d66931ee599732344f85beec94b74140
I20260927 16:39:19.052302 8277 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38577
I20260927 16:39:19.052451 8277 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:19.052909 8277 heartbeater.cc:511] Master 127.0.98.62:38577 requested a full tablet report, sending...
I20260927 16:39:19.053841 7934 ts_manager.cc:194] Registered new tserver with Master: 34362613bae544069a507d0f45be0de5 (127.0.98.2:40193)
I20260927 16:39:19.054483 7934 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35913
I20260927 16:39:19.061684 8283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:19.054201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38577 (local address 127.0.98.2:55589)
0927 16:39:19.054478 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:19.054493 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:19.054618 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:19.054994 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:19.054996 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:19.055182 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:19.055325 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:19.055337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.056344 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.056350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:19.057036 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:19.057044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.057481 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.057484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:19.057569 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:19.058011 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:19.058022 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:19.058292 (+ 270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:19.060281 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:19.060284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:19.060285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:19.060467 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:19.060763 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:19.060765 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:19.060766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:19.060833 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:19.061333 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:19.061335 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:19.061401 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 16:39:19.061476 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 16:39:19.061568 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":106,"threads_started":1}
I20260927 16:39:19.061841 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:19.054507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55589 (local address 127.0.98.62:38577)
0927 16:39:19.054616 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:19.054621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:19.054636 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:39:19.054698 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:19.054701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:19.054741 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:19.054847 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:19.055466 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.056194 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.057155 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.057373 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.057479 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:19.058425 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:19.058432 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:19.058438 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:19.058468 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:19.060175 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:19.060565 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:19.060571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:19.060574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:19.060666 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:19.060932 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:19.060938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:19.060942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:19.061233 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:19.061351 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:19.061558 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 16:39:19.061677 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 16:39:19.081561 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38577
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:19.185484 8286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:19.185747 8286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:19.185856 8286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:19.189801 8286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:19.189883 8286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:19.189952 8286 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 16:39:19.190027 8286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:39:19.195633 8286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:19.196581 8286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:19.197499 8286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:19.206454 8294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:19.206512 8292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:19.206456 8291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:19.206664 8286 server_base.cc:1034] running on GCE node
I20260927 16:39:19.207077 8286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:19.207646 8286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:19.208823 8286 hybrid_clock.cc:648] HybridClock initialized: now 1790527159208809 us; error 25 us; skew 500 ppm
Sep 27 16:39:19 dist-test-slave-z40b krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527159, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:19.211369 8286 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 16:39:19.212107 8286 webserver.cc:460] Webserver started at http://127.0.98.3:40103/ using document root <none> and password file <none>
I20260927 16:39:19.212620 8286 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:19.212702 8286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:19.212889 8286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:19.214828 8286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-2/data/instance:
uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad"
format_stamp: "Formatted at 2026-09-27 16:39:19 on dist-test-slave-z40b"
server_key: "02715d942755043a37e972bda3c4cf61"
server_key_iv: "112fde6d9b619e684dce5dce52f3bda1"
server_key_version: "encryptionkey@0"
I20260927 16:39:19.215394 8286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad"
format_stamp: "Formatted at 2026-09-27 16:39:19 on dist-test-slave-z40b"
server_key: "02715d942755043a37e972bda3c4cf61"
server_key_iv: "112fde6d9b619e684dce5dce52f3bda1"
server_key_version: "encryptionkey@0"
I20260927 16:39:19.218708 8286 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:39:19.220943 8301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:19.221987 8286 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:39:19.222105 8286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad"
format_stamp: "Formatted at 2026-09-27 16:39:19 on dist-test-slave-z40b"
server_key: "02715d942755043a37e972bda3c4cf61"
server_key_iv: "112fde6d9b619e684dce5dce52f3bda1"
server_key_version: "encryptionkey@0"
I20260927 16:39:19.222198 8286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:19.232872 8286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:19.235173 8286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:19.235365 8286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:19.236510 8286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:19.236574 8286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:19.236608 8286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:19.236670 8286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 16:39:19 dist-test-slave-z40b krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527159, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 16:39:19.247282 8286 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33195
I20260927 16:39:19.247622 8416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33195 every 8 connection(s)
I20260927 16:39:19.248313 8286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:39:19.250200 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:19.237383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58147 (local address 127.0.98.62:38577)
0927 16:39:19.237503 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:19.237506 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:19.238416 (+ 910us) server_negotiation.cc:407] Connection header received
0927 16:39:19.239505 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:19.239508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:19.239563 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:19.239653 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:19.241182 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.242038 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.243274 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.243665 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.243799 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:19.246404 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:19.246408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:19.246411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:19.246435 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:19.248154 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:19.248700 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:19.248706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:19.248709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:19.248798 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:19.249208 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:19.249211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:19.249213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:19.249391 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:19.249497 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:19.249938 (+ 441us) server_negotiation.cc:299] Negotiation successful
0927 16:39:19.250044 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:39:19.250999 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:19.237862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38577 (local address 127.0.98.3:58147)
0927 16:39:19.238287 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:19.238322 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:19.239260 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:19.239863 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:19.239874 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:19.240321 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:19.240992 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:19.241014 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.242152 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.242155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:19.243114 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:19.243123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.243872 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.243878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:19.244102 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:19.244739 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:19.244755 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:19.246256 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:19.248257 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:19.248261 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:19.248270 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:19.248575 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:19.248927 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:19.248934 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:19.248937 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:19.249093 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:19.249493 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:19.249498 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:19.249722 (+ 224us) client_negotiation.cc:769] Sending connection context
0927 16:39:19.249899 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 16:39:19.250168 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":120,"threads_started":1}
I20260927 16:39:19.252158 8417 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38577
I20260927 16:39:19.252285 8417 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:19.252880 8417 heartbeater.cc:511] Master 127.0.98.62:38577 requested a full tablet report, sending...
I20260927 16:39:19.253857 7933 ts_manager.cc:194] Registered new tserver with Master: ae2d374cfe354e94b9d9ccc1fee3a5ad (127.0.98.3:33195)
I20260927 16:39:19.254493 7933 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58147
I20260927 16:39:19.257195 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 8286
I20260927 16:39:19.257339 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:39:19.257710 392 external_mini_cluster.cc:1428] Setting key 285b77be0d7f2e101dc3589789eee54b
I20260927 16:39:19.259876 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:39:19.261175 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:19.254172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38577 (local address 127.0.98.3:46199)
0927 16:39:19.254463 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:19.254474 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:19.254548 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:19.254789 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:19.254792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:19.254983 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:19.255110 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:19.255117 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.255933 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.255937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:19.256642 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:19.256651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.257086 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.257090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:19.257184 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:19.257616 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:19.257630 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:19.257987 (+ 357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:19.259870 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:19.259873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:19.259874 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:19.260090 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:19.260350 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:19.260352 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:19.260353 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:19.260420 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:19.260826 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:19.260829 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:19.260894 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 16:39:19.260970 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 16:39:19.261061 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":108,"threads_started":1}
I20260927 16:39:19.261530 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:19.254234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46199 (local address 127.0.98.62:38577)
0927 16:39:19.254335 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:19.254339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:19.254543 (+ 204us) server_negotiation.cc:407] Connection header received
0927 16:39:19.254646 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:19.254649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:19.254686 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:19.254760 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:19.255261 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.255806 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.256738 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.256985 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.257052 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:19.258122 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:19.258125 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:19.258128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:19.258170 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:19.259740 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:19.260174 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:19.260178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:19.260179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:19.260256 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:19.260501 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:19.260504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:19.260505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:19.260738 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:19.260835 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:19.261054 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 16:39:19.261175 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 27 16:39:19 dist-test-slave-z40b krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527159, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 16:39:19 dist-test-slave-z40b krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:19 dist-test-slave-z40b krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 16:39:19.282723 7983 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:19.277003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34360 (local address 127.0.98.62:38577)
0927 16:39:19.277136 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:19.277138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:19.277148 (+ 10us) server_negotiation.cc:407] Connection header received
0927 16:39:19.277176 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:19.277178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:19.277208 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:19.277291 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:19.277945 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.278438 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.279324 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.279534 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.279639 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:19.282573 (+ 2934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34360: BlockingRecv error: recv got EOF from 127.0.0.1:34360 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 16:39:19 dist-test-slave-z40b krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527159, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 16:39:19.291399 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:19.283992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34374 (local address 127.0.98.62:38577)
0927 16:39:19.284114 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:19.284118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:19.284237 (+ 119us) server_negotiation.cc:407] Connection header received
0927 16:39:19.284379 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:19.284382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:19.284426 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:19.284507 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:19.285068 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.285540 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.286317 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.286530 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.286630 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:19.288664 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:19.288670 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:19.288677 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:19.288697 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:19.289798 (+ 1101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:19.290209 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:19.290213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:19.290215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:19.290290 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:19.290564 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:19.290567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:19.290569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:19.290723 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:19.290862 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:19.291083 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 16:39:19.291188 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:39:19.293867 7933 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:39:19.296703 7933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:39:19.308815 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:19.303735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40193 (local address 127.0.0.1:58440)
0927 16:39:19.303996 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:19.304036 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:19.304141 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:19.304826 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:19.304832 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:19.304854 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:19.304994 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:19.305003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.306274 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.306280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:19.307132 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:19.307166 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.308323 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.308329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:19.308469 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:19.308498 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 16:39:19.308587 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 16:39:19.308666 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
I20260927 16:39:19.308880 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:19.303860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58440 (local address 127.0.98.2:40193)
0927 16:39:19.304169 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:19.304178 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:19.304218 (+ 40us) server_negotiation.cc:407] Connection header received
0927 16:39:19.304278 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:19.304282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:19.304497 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:19.304650 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:19.305098 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.306165 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.307320 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.308185 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.308323 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:19.308470 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:19.308630 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 16:39:19.308732 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":131,"threads_started":1}
I20260927 16:39:19.309931 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:19.304359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36701 (local address 127.0.0.1:53298)
0927 16:39:19.304589 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:19.304603 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:19.304673 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:19.305606 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:19.305608 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:19.305624 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:19.305700 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:19.305706 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.307343 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.307350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:19.308614 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:19.308623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.309671 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.309674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:19.309743 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:19.309759 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 16:39:19.309796 (+ 37us) client_negotiation.cc:240] Negotiation successful
0927 16:39:19.309840 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":42,"thread_start_us":89,"threads_started":1}
I20260927 16:39:19.310580 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:19.304578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53298 (local address 127.0.98.1:36701)
0927 16:39:19.305168 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:19.305181 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:19.305202 (+ 21us) server_negotiation.cc:407] Connection header received
0927 16:39:19.305269 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:19.305274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:19.305458 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:19.305873 (+ 415us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:19.306005 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.307181 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.308740 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.309552 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.310122 (+ 570us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:19.310286 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:19.310348 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 16:39:19.310431 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":98,"threads_started":1}
I20260927 16:39:19.310787 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:19.304653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33195 (local address 127.0.0.1:53644)
0927 16:39:19.304845 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:19.304856 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:19.304922 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:19.305955 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:19.305957 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:19.305975 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:19.306048 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:19.306053 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.307362 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.307365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:19.309425 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:19.309435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.310379 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.310382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:19.310467 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:19.310488 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 16:39:19.310569 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 16:39:19.310662 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":884,"thread_start_us":70,"threads_started":1}
I20260927 16:39:19.310822 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:19.305022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53644 (local address 127.0.98.3:33195)
0927 16:39:19.305494 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:19.305506 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:19.305529 (+ 23us) server_negotiation.cc:407] Connection header received
0927 16:39:19.305600 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:19.305604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:19.305805 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:19.305947 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:19.306161 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.307255 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.309548 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.310257 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.310376 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:19.310511 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:19.310612 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 16:39:19.310691 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":100,"threads_started":1}
I20260927 16:39:19.311757 8211 tablet_service.cc:1467] Processing CreateTablet for tablet 4b67a68f580b4ce3927e41183d4f05b2 (DEFAULT_TABLE table=test-table [id=e6192e32b7ae45528febcd2e8f4f96be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:39:19.312794 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b67a68f580b4ce3927e41183d4f05b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:19.313215 8071 tablet_service.cc:1467] Processing CreateTablet for tablet 4b67a68f580b4ce3927e41183d4f05b2 (DEFAULT_TABLE table=test-table [id=e6192e32b7ae45528febcd2e8f4f96be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:39:19.313418 8351 tablet_service.cc:1467] Processing CreateTablet for tablet 4b67a68f580b4ce3927e41183d4f05b2 (DEFAULT_TABLE table=test-table [id=e6192e32b7ae45528febcd2e8f4f96be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:39:19.314000 8071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b67a68f580b4ce3927e41183d4f05b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:19.314268 8351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b67a68f580b4ce3927e41183d4f05b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:19.320868 8445 tablet_bootstrap.cc:492] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec: Bootstrap starting.
I20260927 16:39:19.321054 8447 tablet_bootstrap.cc:492] T 4b67a68f580b4ce3927e41183d4f05b2 P ae2d374cfe354e94b9d9ccc1fee3a5ad: Bootstrap starting.
I20260927 16:39:19.321259 8446 tablet_bootstrap.cc:492] T 4b67a68f580b4ce3927e41183d4f05b2 P 34362613bae544069a507d0f45be0de5: Bootstrap starting.
I20260927 16:39:19.322697 8445 tablet_bootstrap.cc:654] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:19.323123 8447 tablet_bootstrap.cc:654] T 4b67a68f580b4ce3927e41183d4f05b2 P ae2d374cfe354e94b9d9ccc1fee3a5ad: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:19.323123 8446 tablet_bootstrap.cc:654] T 4b67a68f580b4ce3927e41183d4f05b2 P 34362613bae544069a507d0f45be0de5: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:19.323469 8445 log.cc:824] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:19.323963 8446 log.cc:824] T 4b67a68f580b4ce3927e41183d4f05b2 P 34362613bae544069a507d0f45be0de5: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:19.323963 8447 log.cc:824] T 4b67a68f580b4ce3927e41183d4f05b2 P ae2d374cfe354e94b9d9ccc1fee3a5ad: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:19.325345 8445 tablet_bootstrap.cc:492] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec: No bootstrap required, opened a new log
I20260927 16:39:19.325562 8445 ts_tablet_manager.cc:1397] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:39:19.325888 8446 tablet_bootstrap.cc:492] T 4b67a68f580b4ce3927e41183d4f05b2 P 34362613bae544069a507d0f45be0de5: No bootstrap required, opened a new log
I20260927 16:39:19.326082 8446 ts_tablet_manager.cc:1397] T 4b67a68f580b4ce3927e41183d4f05b2 P 34362613bae544069a507d0f45be0de5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:39:19.326316 8447 tablet_bootstrap.cc:492] T 4b67a68f580b4ce3927e41183d4f05b2 P ae2d374cfe354e94b9d9ccc1fee3a5ad: No bootstrap required, opened a new log
I20260927 16:39:19.326516 8447 ts_tablet_manager.cc:1397] T 4b67a68f580b4ce3927e41183d4f05b2 P ae2d374cfe354e94b9d9ccc1fee3a5ad: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:39:19.329336 8447 raft_consensus.cc:348] T 4b67a68f580b4ce3927e41183d4f05b2 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:19.329602 8447 raft_consensus.cc:374] T 4b67a68f580b4ce3927e41183d4f05b2 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:19.329471 8445 raft_consensus.cc:348] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:19.329684 8447 raft_consensus.cc:729] T 4b67a68f580b4ce3927e41183d4f05b2 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae2d374cfe354e94b9d9ccc1fee3a5ad, State: Initialized, Role: FOLLOWER
I20260927 16:39:19.329737 8445 raft_consensus.cc:374] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:19.329840 8445 raft_consensus.cc:729] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad11280495cd4070af56c658de93b7ec, State: Initialized, Role: FOLLOWER
I20260927 16:39:19.329814 8446 raft_consensus.cc:348] T 4b67a68f580b4ce3927e41183d4f05b2 P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:19.330032 8446 raft_consensus.cc:374] T 4b67a68f580b4ce3927e41183d4f05b2 P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:19.330139 8446 raft_consensus.cc:729] T 4b67a68f580b4ce3927e41183d4f05b2 P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34362613bae544069a507d0f45be0de5, State: Initialized, Role: FOLLOWER
I20260927 16:39:19.330161 8447 consensus_queue.cc:260] T 4b67a68f580b4ce3927e41183d4f05b2 P ae2d374cfe354e94b9d9ccc1fee3a5ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:19.330297 8445 consensus_queue.cc:260] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:19.330492 8446 consensus_queue.cc:260] T 4b67a68f580b4ce3927e41183d4f05b2 P 34362613bae544069a507d0f45be0de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:19.331192 8445 ts_tablet_manager.cc:1428] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 16:39:19.331185 8447 ts_tablet_manager.cc:1428] T 4b67a68f580b4ce3927e41183d4f05b2 P ae2d374cfe354e94b9d9ccc1fee3a5ad: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260927 16:39:19.331257 8446 ts_tablet_manager.cc:1428] T 4b67a68f580b4ce3927e41183d4f05b2 P 34362613bae544069a507d0f45be0de5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 16:39:19.331561 8137 heartbeater.cc:503] Master 127.0.98.62:38577 was elected leader, sending a full tablet report...
I20260927 16:39:19.332494 8277 heartbeater.cc:503] Master 127.0.98.62:38577 was elected leader, sending a full tablet report...
I20260927 16:39:19.333335 8417 heartbeater.cc:503] Master 127.0.98.62:38577 was elected leader, sending a full tablet report...
W20260927 16:39:19.350826 8138 tablet.cc:2369] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:39:19.499212 8418 tablet.cc:2369] T 4b67a68f580b4ce3927e41183d4f05b2 P ae2d374cfe354e94b9d9ccc1fee3a5ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:39:19.549391 8278 tablet.cc:2369] T 4b67a68f580b4ce3927e41183d4f05b2 P 34362613bae544069a507d0f45be0de5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:39:19.691334 8452 raft_consensus.cc:482] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:39:19.691593 8452 raft_consensus.cc:504] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:19.692747 8452 leader_election.cc:288] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34362613bae544069a507d0f45be0de5 (127.0.98.2:40193), ae2d374cfe354e94b9d9ccc1fee3a5ad (127.0.98.3:33195)
I20260927 16:39:19.697480 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:19.693521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38681 (local address 127.0.98.3:33195)
0927 16:39:19.693646 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:19.693652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:19.693828 (+ 176us) server_negotiation.cc:407] Connection header received
0927 16:39:19.693909 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:19.693911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:19.693948 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:19.694037 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:19.694292 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.694997 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.696294 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.696834 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.696957 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:19.697018 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:19.697287 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 16:39:19.697366 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 16:39:19.697480 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:19.693116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39833 (local address 127.0.98.2:40193)
0927 16:39:19.693260 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:19.693265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:19.693400 (+ 135us) server_negotiation.cc:407] Connection header received
0927 16:39:19.693566 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:19.693569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:19.693618 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:19.693702 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:19.694031 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.694814 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.696294 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.696842 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.696957 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:19.697018 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:19.697287 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 16:39:19.697357 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 16:39:19.697537 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:19.693399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33195 (local address 127.0.98.1:38681)
0927 16:39:19.693737 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:19.693750 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:19.693830 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:19.694070 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:19.694072 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:19.694088 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:19.694170 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:19.694174 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.695087 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.695090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:19.696177 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:19.696188 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.696976 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.696980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:19.697138 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:19.697179 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 16:39:19.697255 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 16:39:19.697366 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":118,"thread_start_us":58,"threads_started":1}
I20260927 16:39:19.697535 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:19.693021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40193 (local address 127.0.98.1:39833)
0927 16:39:19.693278 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:19.693295 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:19.693453 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:19.693737 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:19.693742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:19.693770 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:19.693915 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:19.693922 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.694926 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.694929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:19.696180 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:19.696188 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.696976 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.696980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:19.697135 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:19.697188 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 16:39:19.697255 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 16:39:19.697361 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":71,"thread_start_us":76,"threads_started":1}
I20260927 16:39:19.698186 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b67a68f580b4ce3927e41183d4f05b2" candidate_uuid: "ad11280495cd4070af56c658de93b7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" is_pre_election: true
I20260927 16:39:19.698181 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b67a68f580b4ce3927e41183d4f05b2" candidate_uuid: "ad11280495cd4070af56c658de93b7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34362613bae544069a507d0f45be0de5" is_pre_election: true
I20260927 16:39:19.698472 8231 raft_consensus.cc:2454] T 4b67a68f580b4ce3927e41183d4f05b2 P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad11280495cd4070af56c658de93b7ec in term 0.
I20260927 16:39:19.698472 8371 raft_consensus.cc:2454] T 4b67a68f580b4ce3927e41183d4f05b2 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad11280495cd4070af56c658de93b7ec in term 0.
I20260927 16:39:19.698963 8022 leader_election.cc:302] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34362613bae544069a507d0f45be0de5, ad11280495cd4070af56c658de93b7ec; no voters:
I20260927 16:39:19.699179 8452 raft_consensus.cc:2781] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:39:19.699303 8452 raft_consensus.cc:482] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:39:19.699364 8452 raft_consensus.cc:3037] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:19.700361 8452 raft_consensus.cc:504] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:19.700719 8452 leader_election.cc:288] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec [CANDIDATE]: Term 1 election: Requested vote from peers 34362613bae544069a507d0f45be0de5 (127.0.98.2:40193), ae2d374cfe354e94b9d9ccc1fee3a5ad (127.0.98.3:33195)
I20260927 16:39:19.701030 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b67a68f580b4ce3927e41183d4f05b2" candidate_uuid: "ad11280495cd4070af56c658de93b7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34362613bae544069a507d0f45be0de5"
I20260927 16:39:19.701120 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b67a68f580b4ce3927e41183d4f05b2" candidate_uuid: "ad11280495cd4070af56c658de93b7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad"
I20260927 16:39:19.701123 8231 raft_consensus.cc:3037] T 4b67a68f580b4ce3927e41183d4f05b2 P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:19.701236 8371 raft_consensus.cc:3037] T 4b67a68f580b4ce3927e41183d4f05b2 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:19.702152 8231 raft_consensus.cc:2454] T 4b67a68f580b4ce3927e41183d4f05b2 P 34362613bae544069a507d0f45be0de5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad11280495cd4070af56c658de93b7ec in term 1.
I20260927 16:39:19.702453 8022 leader_election.cc:302] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34362613bae544069a507d0f45be0de5, ad11280495cd4070af56c658de93b7ec; no voters:
I20260927 16:39:19.702600 8452 raft_consensus.cc:2781] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:39:19.702765 8452 raft_consensus.cc:686] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec [term 1 LEADER]: Becoming Leader. State: Replica: ad11280495cd4070af56c658de93b7ec, State: Running, Role: LEADER
I20260927 16:39:19.702857 8371 raft_consensus.cc:2454] T 4b67a68f580b4ce3927e41183d4f05b2 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad11280495cd4070af56c658de93b7ec in term 1.
I20260927 16:39:19.703125 8452 consensus_queue.cc:237] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:19.706151 7932 catalog_manager.cc:5495] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec reported cstate change: term changed from 0 to 1, leader changed from <none> to ad11280495cd4070af56c658de93b7ec (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ad11280495cd4070af56c658de93b7ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } health_report { overall_health: UNKNOWN } } }
I20260927 16:39:19.756572 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:19.753329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53300 (local address 127.0.98.1:36701)
0927 16:39:19.753451 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:19.753455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:19.753467 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:39:19.753580 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:19.753583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:19.753636 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:19.753721 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:39:19.753914 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.754397 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.755264 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.755476 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.755548 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:19.755867 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:39:19.756257 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:39:19.756346 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:19.756427 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 16:39:19.756470 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:39:19.761981 8231 raft_consensus.cc:1261] T 4b67a68f580b4ce3927e41183d4f05b2 P 34362613bae544069a507d0f45be0de5 [term 1 FOLLOWER]: Refusing update from remote peer ad11280495cd4070af56c658de93b7ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:39:19.761981 8371 raft_consensus.cc:1261] T 4b67a68f580b4ce3927e41183d4f05b2 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 1 FOLLOWER]: Refusing update from remote peer ad11280495cd4070af56c658de93b7ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:39:19.762645 8452 consensus_queue.cc:1035] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:39:19.762873 8456 consensus_queue.cc:1035] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:39:19.770313 8461 mvcc.cc:205] Tried to move back new op lower bound from 7333999246378352640 to 7333999246146469888. Current Snapshot: MvccSnapshot[applied={T|T < 7333999246378352640}]
I20260927 16:39:19.777820 7934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:39:19.777973 7934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:39:19.780205 7934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ba0e51bba58480d8956cbc6689b1379: Sending DeleteTablet for 3 replicas of tablet 4b67a68f580b4ce3927e41183d4f05b2
I20260927 16:39:19.780848 8211 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b67a68f580b4ce3927e41183d4f05b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:39:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58440
I20260927 16:39:19.780949 8351 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b67a68f580b4ce3927e41183d4f05b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:39:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53644
I20260927 16:39:19.781353 8469 tablet_replica.cc:330] T 4b67a68f580b4ce3927e41183d4f05b2 P 34362613bae544069a507d0f45be0de5: stopping tablet replica
I20260927 16:39:19.781409 8470 tablet_replica.cc:330] T 4b67a68f580b4ce3927e41183d4f05b2 P ae2d374cfe354e94b9d9ccc1fee3a5ad: stopping tablet replica
I20260927 16:39:19.781495 8071 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b67a68f580b4ce3927e41183d4f05b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:39:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53298
I20260927 16:39:19.781659 8470 raft_consensus.cc:2229] T 4b67a68f580b4ce3927e41183d4f05b2 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:39:19.781661 8469 raft_consensus.cc:2229] T 4b67a68f580b4ce3927e41183d4f05b2 P 34362613bae544069a507d0f45be0de5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:39:19.781911 8469 raft_consensus.cc:2258] T 4b67a68f580b4ce3927e41183d4f05b2 P 34362613bae544069a507d0f45be0de5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:39:19.781911 8470 raft_consensus.cc:2258] T 4b67a68f580b4ce3927e41183d4f05b2 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:39:19.781914 7932 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:39:19.782300 7932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:39:19.783181 8469 ts_tablet_manager.cc:1905] T 4b67a68f580b4ce3927e41183d4f05b2 P 34362613bae544069a507d0f45be0de5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:39:19.783181 8470 ts_tablet_manager.cc:1905] T 4b67a68f580b4ce3927e41183d4f05b2 P ae2d374cfe354e94b9d9ccc1fee3a5ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:39:19.786127 8469 ts_tablet_manager.cc:1918] T 4b67a68f580b4ce3927e41183d4f05b2 P 34362613bae544069a507d0f45be0de5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:39:19.786278 8469 log.cc:1196] T 4b67a68f580b4ce3927e41183d4f05b2 P 34362613bae544069a507d0f45be0de5: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-1/wal/wals/4b67a68f580b4ce3927e41183d4f05b2
I20260927 16:39:19.786640 8470 ts_tablet_manager.cc:1918] T 4b67a68f580b4ce3927e41183d4f05b2 P ae2d374cfe354e94b9d9ccc1fee3a5ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:39:19.786666 8469 ts_tablet_manager.cc:1939] T 4b67a68f580b4ce3927e41183d4f05b2 P 34362613bae544069a507d0f45be0de5: Deleting consensus metadata
I20260927 16:39:19.786898 8471 tablet_replica.cc:330] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec: stopping tablet replica
I20260927 16:39:19.787201 8471 raft_consensus.cc:2229] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:39:19.787375 8470 log.cc:1196] T 4b67a68f580b4ce3927e41183d4f05b2 P ae2d374cfe354e94b9d9ccc1fee3a5ad: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-2/wal/wals/4b67a68f580b4ce3927e41183d4f05b2
I20260927 16:39:19.787444 8211 tablet_service.cc:1467] Processing CreateTablet for tablet aec54c64dce5482388d02b7dc0def7c2 (DEFAULT_TABLE table=test-table [id=97d30478c63c4a21b44dc04ffd8cae52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:39:19.787683 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aec54c64dce5482388d02b7dc0def7c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:19.787714 8470 ts_tablet_manager.cc:1939] T 4b67a68f580b4ce3927e41183d4f05b2 P ae2d374cfe354e94b9d9ccc1fee3a5ad: Deleting consensus metadata
I20260927 16:39:19.787820 7918 catalog_manager.cc:4841] TS 34362613bae544069a507d0f45be0de5 (127.0.98.2:40193): tablet 4b67a68f580b4ce3927e41183d4f05b2 (table test-table [id=e6192e32b7ae45528febcd2e8f4f96be]) successfully deleted
I20260927 16:39:19.788664 7918 catalog_manager.cc:4841] TS ae2d374cfe354e94b9d9ccc1fee3a5ad (127.0.98.3:33195): tablet 4b67a68f580b4ce3927e41183d4f05b2 (table test-table [id=e6192e32b7ae45528febcd2e8f4f96be]) successfully deleted
I20260927 16:39:19.788866 8471 raft_consensus.cc:2258] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:39:19.789726 8446 tablet_bootstrap.cc:492] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5: Bootstrap starting.
I20260927 16:39:19.789870 8351 tablet_service.cc:1467] Processing CreateTablet for tablet aec54c64dce5482388d02b7dc0def7c2 (DEFAULT_TABLE table=test-table [id=97d30478c63c4a21b44dc04ffd8cae52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:39:19.789937 8471 ts_tablet_manager.cc:1905] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:39:19.790091 8351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aec54c64dce5482388d02b7dc0def7c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:19.790637 8446 tablet_bootstrap.cc:654] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:19.792265 8447 tablet_bootstrap.cc:492] T aec54c64dce5482388d02b7dc0def7c2 P ae2d374cfe354e94b9d9ccc1fee3a5ad: Bootstrap starting.
I20260927 16:39:19.792392 8446 tablet_bootstrap.cc:492] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5: No bootstrap required, opened a new log
I20260927 16:39:19.792459 8446 ts_tablet_manager.cc:1397] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 16:39:19.792712 8471 ts_tablet_manager.cc:1918] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:39:19.792784 8471 log.cc:1196] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-0/wal/wals/4b67a68f580b4ce3927e41183d4f05b2
I20260927 16:39:19.792922 8446 raft_consensus.cc:348] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:19.793032 8446 raft_consensus.cc:374] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:19.793067 8447 tablet_bootstrap.cc:654] T aec54c64dce5482388d02b7dc0def7c2 P ae2d374cfe354e94b9d9ccc1fee3a5ad: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:19.793069 8471 ts_tablet_manager.cc:1939] T 4b67a68f580b4ce3927e41183d4f05b2 P ad11280495cd4070af56c658de93b7ec: Deleting consensus metadata
I20260927 16:39:19.793081 8446 raft_consensus.cc:729] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34362613bae544069a507d0f45be0de5, State: Initialized, Role: FOLLOWER
I20260927 16:39:19.793207 8446 consensus_queue.cc:260] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:19.793435 8446 ts_tablet_manager.cc:1428] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:39:19.793977 7918 catalog_manager.cc:4841] TS ad11280495cd4070af56c658de93b7ec (127.0.98.1:36701): tablet 4b67a68f580b4ce3927e41183d4f05b2 (table test-table [id=e6192e32b7ae45528febcd2e8f4f96be]) successfully deleted
I20260927 16:39:19.794200 8071 tablet_service.cc:1467] Processing CreateTablet for tablet aec54c64dce5482388d02b7dc0def7c2 (DEFAULT_TABLE table=test-table [id=97d30478c63c4a21b44dc04ffd8cae52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:39:19.794422 8071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aec54c64dce5482388d02b7dc0def7c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:19.796053 8447 tablet_bootstrap.cc:492] T aec54c64dce5482388d02b7dc0def7c2 P ae2d374cfe354e94b9d9ccc1fee3a5ad: No bootstrap required, opened a new log
I20260927 16:39:19.796150 8447 ts_tablet_manager.cc:1397] T aec54c64dce5482388d02b7dc0def7c2 P ae2d374cfe354e94b9d9ccc1fee3a5ad: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260927 16:39:19.796340 8445 tablet_bootstrap.cc:492] T aec54c64dce5482388d02b7dc0def7c2 P ad11280495cd4070af56c658de93b7ec: Bootstrap starting.
I20260927 16:39:19.796607 8447 raft_consensus.cc:348] T aec54c64dce5482388d02b7dc0def7c2 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:19.796720 8447 raft_consensus.cc:374] T aec54c64dce5482388d02b7dc0def7c2 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:19.796752 8447 raft_consensus.cc:729] T aec54c64dce5482388d02b7dc0def7c2 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae2d374cfe354e94b9d9ccc1fee3a5ad, State: Initialized, Role: FOLLOWER
I20260927 16:39:19.796878 8447 consensus_queue.cc:260] T aec54c64dce5482388d02b7dc0def7c2 P ae2d374cfe354e94b9d9ccc1fee3a5ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:19.797106 8447 ts_tablet_manager.cc:1428] T aec54c64dce5482388d02b7dc0def7c2 P ae2d374cfe354e94b9d9ccc1fee3a5ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 16:39:19.797242 8445 tablet_bootstrap.cc:654] T aec54c64dce5482388d02b7dc0def7c2 P ad11280495cd4070af56c658de93b7ec: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:19.798619 8445 tablet_bootstrap.cc:492] T aec54c64dce5482388d02b7dc0def7c2 P ad11280495cd4070af56c658de93b7ec: No bootstrap required, opened a new log
I20260927 16:39:19.798692 8445 ts_tablet_manager.cc:1397] T aec54c64dce5482388d02b7dc0def7c2 P ad11280495cd4070af56c658de93b7ec: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 16:39:19.799062 8445 raft_consensus.cc:348] T aec54c64dce5482388d02b7dc0def7c2 P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:19.799177 8445 raft_consensus.cc:374] T aec54c64dce5482388d02b7dc0def7c2 P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:19.799228 8445 raft_consensus.cc:729] T aec54c64dce5482388d02b7dc0def7c2 P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad11280495cd4070af56c658de93b7ec, State: Initialized, Role: FOLLOWER
I20260927 16:39:19.799342 8445 consensus_queue.cc:260] T aec54c64dce5482388d02b7dc0def7c2 P ad11280495cd4070af56c658de93b7ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:19.799607 8445 ts_tablet_manager.cc:1428] T aec54c64dce5482388d02b7dc0def7c2 P ad11280495cd4070af56c658de93b7ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 16:39:19.953294 8453 raft_consensus.cc:482] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:39:19.953421 8453 raft_consensus.cc:504] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:19.954412 8453 leader_election.cc:288] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad11280495cd4070af56c658de93b7ec (127.0.98.1:36701), ae2d374cfe354e94b9d9ccc1fee3a5ad (127.0.98.3:33195)
I20260927 16:39:19.959543 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:19.955185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58229 (local address 127.0.98.3:33195)
0927 16:39:19.955332 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:19.955347 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:19.955400 (+ 53us) server_negotiation.cc:407] Connection header received
0927 16:39:19.955547 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:19.955550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:19.955601 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:19.955671 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:19.955923 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.956595 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.958268 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.958840 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.958956 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:19.959016 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:19.959351 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 16:39:19.959422 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 16:39:19.959543 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:19.954740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45351 (local address 127.0.98.1:36701)
0927 16:39:19.954899 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:19.954903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:19.955066 (+ 163us) server_negotiation.cc:407] Connection header received
0927 16:39:19.955250 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:19.955254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:19.955309 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:19.955427 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:19.955756 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.956512 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.958268 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.958840 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.958956 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:19.959017 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:19.959351 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 16:39:19.959419 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 16:39:19.959586 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:19.954703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36701 (local address 127.0.98.2:45351)
0927 16:39:19.954968 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:19.954979 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:19.955111 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:19.955457 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:19.955462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:19.955480 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:19.955636 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:19.955644 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.956614 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.956616 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:19.958144 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:19.958155 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.958997 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.959001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:19.959186 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:19.959246 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 16:39:19.959321 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 16:39:19.959405 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":169,"thread_start_us":89,"threads_started":1}
I20260927 16:39:19.959928 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:19.955067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33195 (local address 127.0.98.2:58229)
0927 16:39:19.955331 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:19.955343 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:19.955431 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:19.955683 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:19.955688 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:19.955708 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:19.955817 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:19.955821 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.956687 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.956690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:19.958147 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:19.958155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:19.958997 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:19.959001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:19.959166 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:19.959249 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 16:39:19.959321 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 16:39:19.959410 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":236,"thread_start_us":91,"threads_started":1}
I20260927 16:39:19.960165 8075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec54c64dce5482388d02b7dc0def7c2" candidate_uuid: "34362613bae544069a507d0f45be0de5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad11280495cd4070af56c658de93b7ec" is_pre_election: true
I20260927 16:39:19.960402 8075 raft_consensus.cc:2454] T aec54c64dce5482388d02b7dc0def7c2 P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34362613bae544069a507d0f45be0de5 in term 0.
I20260927 16:39:19.960844 8162 leader_election.cc:302] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34362613bae544069a507d0f45be0de5, ad11280495cd4070af56c658de93b7ec; no voters:
I20260927 16:39:19.960979 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec54c64dce5482388d02b7dc0def7c2" candidate_uuid: "34362613bae544069a507d0f45be0de5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" is_pre_election: true
I20260927 16:39:19.961102 8453 raft_consensus.cc:2781] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:39:19.961140 8371 raft_consensus.cc:2454] T aec54c64dce5482388d02b7dc0def7c2 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34362613bae544069a507d0f45be0de5 in term 0.
I20260927 16:39:19.961161 8453 raft_consensus.cc:482] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:39:19.961205 8453 raft_consensus.cc:3037] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:19.962178 8453 raft_consensus.cc:504] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:19.962448 8453 leader_election.cc:288] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5 [CANDIDATE]: Term 1 election: Requested vote from peers ad11280495cd4070af56c658de93b7ec (127.0.98.1:36701), ae2d374cfe354e94b9d9ccc1fee3a5ad (127.0.98.3:33195)
I20260927 16:39:19.962736 8075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec54c64dce5482388d02b7dc0def7c2" candidate_uuid: "34362613bae544069a507d0f45be0de5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad11280495cd4070af56c658de93b7ec"
I20260927 16:39:19.962869 8075 raft_consensus.cc:3037] T aec54c64dce5482388d02b7dc0def7c2 P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:19.962850 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec54c64dce5482388d02b7dc0def7c2" candidate_uuid: "34362613bae544069a507d0f45be0de5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad"
I20260927 16:39:19.962973 8371 raft_consensus.cc:3037] T aec54c64dce5482388d02b7dc0def7c2 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:19.964058 8371 raft_consensus.cc:2454] T aec54c64dce5482388d02b7dc0def7c2 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34362613bae544069a507d0f45be0de5 in term 1.
I20260927 16:39:19.964058 8075 raft_consensus.cc:2454] T aec54c64dce5482388d02b7dc0def7c2 P ad11280495cd4070af56c658de93b7ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34362613bae544069a507d0f45be0de5 in term 1.
I20260927 16:39:19.964344 8162 leader_election.cc:302] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34362613bae544069a507d0f45be0de5, ae2d374cfe354e94b9d9ccc1fee3a5ad; no voters:
I20260927 16:39:19.964588 8453 raft_consensus.cc:2781] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:39:19.964768 8453 raft_consensus.cc:686] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5 [term 1 LEADER]: Becoming Leader. State: Replica: 34362613bae544069a507d0f45be0de5, State: Running, Role: LEADER
I20260927 16:39:19.965021 8453 consensus_queue.cc:237] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:19.967304 7932 catalog_manager.cc:5495] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34362613bae544069a507d0f45be0de5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "34362613bae544069a507d0f45be0de5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } health_report { overall_health: UNKNOWN } } }
Sep 27 16:39:20 dist-test-slave-z40b krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 16:39:20.008409 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:20.000812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38577 (local address 127.0.0.1:34390)
0927 16:39:20.001082 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:20.001094 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:20.001175 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:20.001400 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:20.001402 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:20.001762 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:20.001852 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:20.001857 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.002546 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.002548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:20.003076 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:20.003082 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.003591 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.003594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:20.003654 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:20.004219 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:20.004231 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:20.005574 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:20.007082 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:20.007089 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:20.007093 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:20.007377 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:20.007609 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:20.007611 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:20.007613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:20.007685 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:20.007958 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:20.007963 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:20.008057 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 16:39:20.008145 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 16:39:20.008251 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":95,"threads_started":1}
I20260927 16:39:20.008442 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:20.000860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34390 (local address 127.0.98.62:38577)
0927 16:39:20.001119 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:20.001126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:20.001140 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:39:20.001253 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:20.001255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:20.001293 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:20.001367 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:20.001946 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.002462 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.003195 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.003491 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.003592 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:20.005686 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:20.005689 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:20.005691 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:20.005710 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:20.006990 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:20.007456 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:20.007458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:20.007460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:20.007532 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:20.007738 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:20.007739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:20.007740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:20.007881 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:20.007948 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:20.008192 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 16:39:20.008306 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":190,"thread_start_us":107,"threads_started":1}
I20260927 16:39:20.011858 7932 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34390:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 16:39:20.012467 7932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:39:20.016963 8211 tablet_service.cc:1467] Processing CreateTablet for tablet 595014329ff44aac8860aaaa550a9820 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=46fcd7f012f04e40ac294a6c7361dc22]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 16:39:20.017074 8351 tablet_service.cc:1467] Processing CreateTablet for tablet 595014329ff44aac8860aaaa550a9820 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=46fcd7f012f04e40ac294a6c7361dc22]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 16:39:20.017074 8071 tablet_service.cc:1467] Processing CreateTablet for tablet 595014329ff44aac8860aaaa550a9820 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=46fcd7f012f04e40ac294a6c7361dc22]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 16:39:20.017336 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595014329ff44aac8860aaaa550a9820. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:20.017433 8351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595014329ff44aac8860aaaa550a9820. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:20.017433 8071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595014329ff44aac8860aaaa550a9820. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:20.019758 8446 tablet_bootstrap.cc:492] T 595014329ff44aac8860aaaa550a9820 P 34362613bae544069a507d0f45be0de5: Bootstrap starting.
I20260927 16:39:20.020136 8445 tablet_bootstrap.cc:492] T 595014329ff44aac8860aaaa550a9820 P ad11280495cd4070af56c658de93b7ec: Bootstrap starting.
I20260927 16:39:20.020685 8446 tablet_bootstrap.cc:654] T 595014329ff44aac8860aaaa550a9820 P 34362613bae544069a507d0f45be0de5: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:20.020980 8445 tablet_bootstrap.cc:654] T 595014329ff44aac8860aaaa550a9820 P ad11280495cd4070af56c658de93b7ec: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:20.021384 8447 tablet_bootstrap.cc:492] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad: Bootstrap starting.
I20260927 16:39:20.022200 8445 tablet_bootstrap.cc:492] T 595014329ff44aac8860aaaa550a9820 P ad11280495cd4070af56c658de93b7ec: No bootstrap required, opened a new log
I20260927 16:39:20.022292 8445 ts_tablet_manager.cc:1397] T 595014329ff44aac8860aaaa550a9820 P ad11280495cd4070af56c658de93b7ec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 16:39:20.022336 8447 tablet_bootstrap.cc:654] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:20.022719 8445 raft_consensus.cc:348] T 595014329ff44aac8860aaaa550a9820 P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:20.022840 8445 raft_consensus.cc:374] T 595014329ff44aac8860aaaa550a9820 P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:20.022881 8445 raft_consensus.cc:729] T 595014329ff44aac8860aaaa550a9820 P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad11280495cd4070af56c658de93b7ec, State: Initialized, Role: FOLLOWER
I20260927 16:39:20.023008 8445 consensus_queue.cc:260] T 595014329ff44aac8860aaaa550a9820 P ad11280495cd4070af56c658de93b7ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:20.023186 8446 tablet_bootstrap.cc:492] T 595014329ff44aac8860aaaa550a9820 P 34362613bae544069a507d0f45be0de5: No bootstrap required, opened a new log
I20260927 16:39:20.023288 8446 ts_tablet_manager.cc:1397] T 595014329ff44aac8860aaaa550a9820 P 34362613bae544069a507d0f45be0de5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 16:39:20.023309 8445 ts_tablet_manager.cc:1428] T 595014329ff44aac8860aaaa550a9820 P ad11280495cd4070af56c658de93b7ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:39:20.023350 8465 tablet_replica.cc:439] T 595014329ff44aac8860aaaa550a9820 P ad11280495cd4070af56c658de93b7ec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } } }
I20260927 16:39:20.023499 8465 tablet_replica.cc:442] T 595014329ff44aac8860aaaa550a9820 P ad11280495cd4070af56c658de93b7ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:39:20.023644 8446 raft_consensus.cc:348] T 595014329ff44aac8860aaaa550a9820 P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:20.023761 8446 raft_consensus.cc:374] T 595014329ff44aac8860aaaa550a9820 P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:20.023800 8446 raft_consensus.cc:729] T 595014329ff44aac8860aaaa550a9820 P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34362613bae544069a507d0f45be0de5, State: Initialized, Role: FOLLOWER
I20260927 16:39:20.023957 8446 consensus_queue.cc:260] T 595014329ff44aac8860aaaa550a9820 P 34362613bae544069a507d0f45be0de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:20.024219 8446 ts_tablet_manager.cc:1428] T 595014329ff44aac8860aaaa550a9820 P 34362613bae544069a507d0f45be0de5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 16:39:20.024164 8453 tablet_replica.cc:439] T 595014329ff44aac8860aaaa550a9820 P 34362613bae544069a507d0f45be0de5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } } }
I20260927 16:39:20.024353 8453 tablet_replica.cc:442] T 595014329ff44aac8860aaaa550a9820 P 34362613bae544069a507d0f45be0de5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:39:20.024848 8447 tablet_bootstrap.cc:492] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad: No bootstrap required, opened a new log
I20260927 16:39:20.024928 8447 ts_tablet_manager.cc:1397] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260927 16:39:20.025385 8447 raft_consensus.cc:348] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:20.025491 8447 raft_consensus.cc:374] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:20.025527 8447 raft_consensus.cc:729] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae2d374cfe354e94b9d9ccc1fee3a5ad, State: Initialized, Role: FOLLOWER
I20260927 16:39:20.025635 8447 consensus_queue.cc:260] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:20.025928 8447 ts_tablet_manager.cc:1428] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 16:39:20.026203 8451 tablet_replica.cc:439] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } } }
I20260927 16:39:20.026332 8451 tablet_replica.cc:442] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:39:20.320480 8451 raft_consensus.cc:482] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:39:20.320611 8451 raft_consensus.cc:504] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:20.321691 8451 leader_election.cc:288] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad11280495cd4070af56c658de93b7ec (127.0.98.1:36701), 34362613bae544069a507d0f45be0de5 (127.0.98.2:40193)
I20260927 16:39:20.326759 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:20.322449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42323 (local address 127.0.98.2:40193)
0927 16:39:20.322670 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:20.322674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:20.322687 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:39:20.322738 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:20.322740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:20.322775 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:20.322876 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:20.323088 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.323818 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.325429 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.326080 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.326198 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:20.326255 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:20.326561 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 16:39:20.326633 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":62,"threads_started":1}
I20260927 16:39:20.326756 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:20.322012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58393 (local address 127.0.98.1:36701)
0927 16:39:20.322144 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:20.322148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:20.322257 (+ 109us) server_negotiation.cc:407] Connection header received
0927 16:39:20.322419 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:20.322423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:20.322477 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:20.322573 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:20.322878 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.323697 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.325467 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.326080 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.326193 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:20.326254 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:20.326577 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 16:39:20.326645 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:39:20.326797 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:20.322288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40193 (local address 127.0.98.3:42323)
0927 16:39:20.322509 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:20.322519 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:20.322648 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:20.322895 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:20.322897 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:20.322912 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:20.322987 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:20.322991 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.323912 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.323915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:20.325275 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:20.325287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.326217 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.326220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:20.326364 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:20.326431 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 16:39:20.326510 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 16:39:20.326616 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":117,"spinlock_wait_cycles":66432,"thread_start_us":61,"threads_started":1}
I20260927 16:39:20.326808 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:20.321956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36701 (local address 127.0.98.3:58393)
0927 16:39:20.322154 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:20.322169 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:20.322278 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:20.322595 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:20.322598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:20.322636 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:20.322766 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:20.322772 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.323794 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.323796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:20.325277 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:20.325287 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.326216 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.326221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:20.326391 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:20.326436 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 16:39:20.326516 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 16:39:20.326616 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":278,"thread_start_us":72,"threads_started":1}
I20260927 16:39:20.327370 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595014329ff44aac8860aaaa550a9820" candidate_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34362613bae544069a507d0f45be0de5" is_pre_election: true
I20260927 16:39:20.327435 8075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595014329ff44aac8860aaaa550a9820" candidate_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad11280495cd4070af56c658de93b7ec" is_pre_election: true
I20260927 16:39:20.327518 8231 raft_consensus.cc:2454] T 595014329ff44aac8860aaaa550a9820 P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae2d374cfe354e94b9d9ccc1fee3a5ad in term 0.
I20260927 16:39:20.327579 8075 raft_consensus.cc:2454] T 595014329ff44aac8860aaaa550a9820 P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae2d374cfe354e94b9d9ccc1fee3a5ad in term 0.
I20260927 16:39:20.327853 8302 leader_election.cc:302] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34362613bae544069a507d0f45be0de5, ae2d374cfe354e94b9d9ccc1fee3a5ad; no voters:
I20260927 16:39:20.328073 8451 raft_consensus.cc:2781] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:39:20.328133 8451 raft_consensus.cc:482] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:39:20.328176 8451 raft_consensus.cc:3037] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:20.329136 8451 raft_consensus.cc:504] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:20.329500 8451 leader_election.cc:288] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad [CANDIDATE]: Term 1 election: Requested vote from peers ad11280495cd4070af56c658de93b7ec (127.0.98.1:36701), 34362613bae544069a507d0f45be0de5 (127.0.98.2:40193)
I20260927 16:39:20.329803 8075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595014329ff44aac8860aaaa550a9820" candidate_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad11280495cd4070af56c658de93b7ec"
I20260927 16:39:20.329878 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595014329ff44aac8860aaaa550a9820" candidate_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34362613bae544069a507d0f45be0de5"
I20260927 16:39:20.329921 8075 raft_consensus.cc:3037] T 595014329ff44aac8860aaaa550a9820 P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:20.329988 8231 raft_consensus.cc:3037] T 595014329ff44aac8860aaaa550a9820 P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:20.330940 8231 raft_consensus.cc:2454] T 595014329ff44aac8860aaaa550a9820 P 34362613bae544069a507d0f45be0de5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae2d374cfe354e94b9d9ccc1fee3a5ad in term 1.
I20260927 16:39:20.331007 8075 raft_consensus.cc:2454] T 595014329ff44aac8860aaaa550a9820 P ad11280495cd4070af56c658de93b7ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae2d374cfe354e94b9d9ccc1fee3a5ad in term 1.
I20260927 16:39:20.331223 8302 leader_election.cc:302] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34362613bae544069a507d0f45be0de5, ae2d374cfe354e94b9d9ccc1fee3a5ad; no voters:
I20260927 16:39:20.331388 8451 raft_consensus.cc:2781] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:39:20.331534 8451 raft_consensus.cc:686] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 1 LEADER]: Becoming Leader. State: Replica: ae2d374cfe354e94b9d9ccc1fee3a5ad, State: Running, Role: LEADER
I20260927 16:39:20.331790 8451 consensus_queue.cc:237] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:20.332880 8492 tablet_replica.cc:439] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad: TxnStatusTablet state changed. Reason: New leader ae2d374cfe354e94b9d9ccc1fee3a5ad. Latest consensus state: current_term: 1 leader_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } } }
I20260927 16:39:20.332995 8492 tablet_replica.cc:442] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad: This TxnStatusTablet replica's current role is: LEADER
I20260927 16:39:20.333244 8494 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 16:39:20.333403 8494 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 16:39:20.334044 7932 catalog_manager.cc:5495] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad reported cstate change: term changed from 0 to 1, leader changed from <none> to ae2d374cfe354e94b9d9ccc1fee3a5ad (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } health_report { overall_health: HEALTHY } } }
I20260927 16:39:20.386310 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:20.382268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33195 (local address 127.0.0.1:53656)
0927 16:39:20.382394 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:20.382411 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:20.382510 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:20.382836 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:20.382839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:20.382846 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:39:20.382938 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:20.382942 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.383973 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.383976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:20.384676 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:20.384685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.385212 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.385214 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:20.385273 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:20.385311 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:39:20.386069 (+ 758us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:39:20.386082 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 16:39:20.386136 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 16:39:20.386206 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 16:39:20.386327 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:20.382353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53656 (local address 127.0.98.3:33195)
0927 16:39:20.382475 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:20.382479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:20.382494 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:39:20.382653 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:20.382656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:20.382719 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:20.382821 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:39:20.383038 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.383874 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.384780 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.385112 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.385210 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:20.385453 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:39:20.385979 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:39:20.386087 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:20.386150 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 16:39:20.386198 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:39:20.389257 8231 raft_consensus.cc:1261] T 595014329ff44aac8860aaaa550a9820 P 34362613bae544069a507d0f45be0de5 [term 1 FOLLOWER]: Refusing update from remote peer ae2d374cfe354e94b9d9ccc1fee3a5ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:39:20.389462 8075 raft_consensus.cc:1261] T 595014329ff44aac8860aaaa550a9820 P ad11280495cd4070af56c658de93b7ec [term 1 FOLLOWER]: Refusing update from remote peer ae2d374cfe354e94b9d9ccc1fee3a5ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:39:20.389737 8492 consensus_queue.cc:1035] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:39:20.390003 8451 consensus_queue.cc:1035] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:39:20.391249 8453 tablet_replica.cc:439] T 595014329ff44aac8860aaaa550a9820 P 34362613bae544069a507d0f45be0de5: TxnStatusTablet state changed. Reason: New leader ae2d374cfe354e94b9d9ccc1fee3a5ad. Latest consensus state: current_term: 1 leader_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } } }
I20260927 16:39:20.391358 8453 tablet_replica.cc:442] T 595014329ff44aac8860aaaa550a9820 P 34362613bae544069a507d0f45be0de5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:39:20.391739 8465 tablet_replica.cc:439] T 595014329ff44aac8860aaaa550a9820 P ad11280495cd4070af56c658de93b7ec: TxnStatusTablet state changed. Reason: New leader ae2d374cfe354e94b9d9ccc1fee3a5ad. Latest consensus state: current_term: 1 leader_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } } }
I20260927 16:39:20.391840 8465 tablet_replica.cc:442] T 595014329ff44aac8860aaaa550a9820 P ad11280495cd4070af56c658de93b7ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:39:20.393344 8492 tablet_replica.cc:439] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } } }
I20260927 16:39:20.393438 8492 tablet_replica.cc:442] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad: This TxnStatusTablet replica's current role is: LEADER
I20260927 16:39:20.393519 8497 mvcc.cc:205] Tried to move back new op lower bound from 7333999248949719040 to 7333999248721231872. Current Snapshot: MvccSnapshot[applied={T|T < 7333999248949719040}]
I20260927 16:39:20.393669 8465 tablet_replica.cc:439] T 595014329ff44aac8860aaaa550a9820 P ad11280495cd4070af56c658de93b7ec: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } } }
I20260927 16:39:20.393769 8465 tablet_replica.cc:442] T 595014329ff44aac8860aaaa550a9820 P ad11280495cd4070af56c658de93b7ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:39:20.394258 8500 mvcc.cc:205] Tried to move back new op lower bound from 7333999248949719040 to 7333999248721231872. Current Snapshot: MvccSnapshot[applied={T|T < 7333999248949719040}]
I20260927 16:39:20.397848 8451 tablet_replica.cc:439] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } } }
I20260927 16:39:20.397951 8451 tablet_replica.cc:442] T 595014329ff44aac8860aaaa550a9820 P ae2d374cfe354e94b9d9ccc1fee3a5ad: This TxnStatusTablet replica's current role is: LEADER
I20260927 16:39:20.399691 8453 tablet_replica.cc:439] T 595014329ff44aac8860aaaa550a9820 P 34362613bae544069a507d0f45be0de5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } } }
I20260927 16:39:20.399796 8453 tablet_replica.cc:442] T 595014329ff44aac8860aaaa550a9820 P 34362613bae544069a507d0f45be0de5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:39:20.402575 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:20.397972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58446 (local address 127.0.98.2:40193)
0927 16:39:20.398116 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:20.398120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:20.398132 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:39:20.398277 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:20.398281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:20.398337 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:20.398404 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:39:20.398772 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.399548 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.401341 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.401556 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.401635 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:20.401900 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:39:20.402262 (+ 362us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:39:20.402342 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:20.402446 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 16:39:20.402487 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 16:39:20.412297 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:20.408986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40919 (local address 127.0.98.3:33195)
0927 16:39:20.409112 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:20.409115 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:20.409227 (+ 112us) server_negotiation.cc:407] Connection header received
0927 16:39:20.409387 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:20.409390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:20.409426 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:20.409485 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:39:20.409735 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.410166 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.411361 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.411567 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.411642 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:20.411933 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:39:20.411989 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:39:20.412038 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:20.412158 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 16:39:20.412197 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 16:39:20.412361 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:20.408855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33195 (local address 127.0.98.2:40919)
0927 16:39:20.409145 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:20.409159 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:20.409262 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:20.409514 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:20.409517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:20.409524 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:39:20.409646 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:20.409651 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.410257 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.410260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:20.411256 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:20.411268 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.411668 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.411671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:20.411765 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:20.411809 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:39:20.412074 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:39:20.412080 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 16:39:20.412143 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 16:39:20.412216 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":117,"threads_started":1}
I20260927 16:39:20.418452 8371 raft_consensus.cc:1261] T aec54c64dce5482388d02b7dc0def7c2 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 1 FOLLOWER]: Refusing update from remote peer 34362613bae544069a507d0f45be0de5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:39:20.418496 8075 raft_consensus.cc:1261] T aec54c64dce5482388d02b7dc0def7c2 P ad11280495cd4070af56c658de93b7ec [term 1 FOLLOWER]: Refusing update from remote peer 34362613bae544069a507d0f45be0de5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:39:20.418949 8477 consensus_queue.cc:1035] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:39:20.419116 8453 consensus_queue.cc:1035] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:39:20.440886 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:20.437467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58953 (local address 127.0.98.2:40193)
0927 16:39:20.437577 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:20.437581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:20.437771 (+ 190us) server_negotiation.cc:407] Connection header received
0927 16:39:20.437907 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:20.437910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:20.437945 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:20.437997 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:39:20.438294 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.438803 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.439861 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.440072 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.440178 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:20.440483 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:39:20.440545 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:39:20.440624 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:20.440712 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 16:39:20.440754 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 16:39:20.440943 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:20.437344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40193 (local address 127.0.98.3:58953)
0927 16:39:20.437681 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:20.437698 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:20.437806 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:20.438035 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:20.438037 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:20.438046 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:39:20.438156 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:20.438162 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.438917 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.438920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:20.439738 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:20.439747 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.440202 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.440207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:20.440292 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:20.440328 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:39:20.440630 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:39:20.440638 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 16:39:20.440703 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 16:39:20.440777 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":124,"threads_started":1}
I20260927 16:39:20.466617 7932 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:39:20.466789 7932 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:39:20.469043 7932 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ba0e51bba58480d8956cbc6689b1379: Sending DeleteTablet for 3 replicas of tablet aec54c64dce5482388d02b7dc0def7c2
I20260927 16:39:20.469781 8071 tablet_service.cc:1514] Processing DeleteTablet for tablet aec54c64dce5482388d02b7dc0def7c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:39:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53298
I20260927 16:39:20.469900 8211 tablet_service.cc:1514] Processing DeleteTablet for tablet aec54c64dce5482388d02b7dc0def7c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:39:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58440
I20260927 16:39:20.470271 8351 tablet_service.cc:1514] Processing DeleteTablet for tablet aec54c64dce5482388d02b7dc0def7c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:39:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53644
I20260927 16:39:20.473654 8519 tablet_replica.cc:330] T aec54c64dce5482388d02b7dc0def7c2 P ad11280495cd4070af56c658de93b7ec: stopping tablet replica
I20260927 16:39:20.473794 8519 raft_consensus.cc:2229] T aec54c64dce5482388d02b7dc0def7c2 P ad11280495cd4070af56c658de93b7ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:39:20.473912 8519 raft_consensus.cc:2258] T aec54c64dce5482388d02b7dc0def7c2 P ad11280495cd4070af56c658de93b7ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:39:20.474120 8520 tablet_replica.cc:330] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5: stopping tablet replica
I20260927 16:39:20.474228 8520 raft_consensus.cc:2229] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:39:20.474493 8520 raft_consensus.cc:2258] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:39:20.474709 8519 ts_tablet_manager.cc:1905] T aec54c64dce5482388d02b7dc0def7c2 P ad11280495cd4070af56c658de93b7ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:39:20.474745 8521 tablet_replica.cc:330] T aec54c64dce5482388d02b7dc0def7c2 P ae2d374cfe354e94b9d9ccc1fee3a5ad: stopping tablet replica
I20260927 16:39:20.474867 8521 raft_consensus.cc:2229] T aec54c64dce5482388d02b7dc0def7c2 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:39:20.474975 8521 raft_consensus.cc:2258] T aec54c64dce5482388d02b7dc0def7c2 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:39:20.475334 8520 ts_tablet_manager.cc:1905] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:39:20.478135 8519 ts_tablet_manager.cc:1918] T aec54c64dce5482388d02b7dc0def7c2 P ad11280495cd4070af56c658de93b7ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:39:20.478137 8520 ts_tablet_manager.cc:1918] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:39:20.478212 8520 log.cc:1196] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-1/wal/wals/aec54c64dce5482388d02b7dc0def7c2
I20260927 16:39:20.478313 8519 log.cc:1196] T aec54c64dce5482388d02b7dc0def7c2 P ad11280495cd4070af56c658de93b7ec: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-0/wal/wals/aec54c64dce5482388d02b7dc0def7c2
I20260927 16:39:20.478497 8520 ts_tablet_manager.cc:1939] T aec54c64dce5482388d02b7dc0def7c2 P 34362613bae544069a507d0f45be0de5: Deleting consensus metadata
I20260927 16:39:20.478680 8519 ts_tablet_manager.cc:1939] T aec54c64dce5482388d02b7dc0def7c2 P ad11280495cd4070af56c658de93b7ec: Deleting consensus metadata
I20260927 16:39:20.479161 7918 catalog_manager.cc:4841] TS 34362613bae544069a507d0f45be0de5 (127.0.98.2:40193): tablet aec54c64dce5482388d02b7dc0def7c2 (table test-table [id=97d30478c63c4a21b44dc04ffd8cae52]) successfully deleted
I20260927 16:39:20.479346 7918 catalog_manager.cc:4841] TS ad11280495cd4070af56c658de93b7ec (127.0.98.1:36701): tablet aec54c64dce5482388d02b7dc0def7c2 (table test-table [id=97d30478c63c4a21b44dc04ffd8cae52]) successfully deleted
I20260927 16:39:20.480000 8521 ts_tablet_manager.cc:1905] T aec54c64dce5482388d02b7dc0def7c2 P ae2d374cfe354e94b9d9ccc1fee3a5ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:39:20.483103 8521 ts_tablet_manager.cc:1918] T aec54c64dce5482388d02b7dc0def7c2 P ae2d374cfe354e94b9d9ccc1fee3a5ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:39:20.483199 8521 log.cc:1196] T aec54c64dce5482388d02b7dc0def7c2 P ae2d374cfe354e94b9d9ccc1fee3a5ad: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-2/wal/wals/aec54c64dce5482388d02b7dc0def7c2
I20260927 16:39:20.483511 8521 ts_tablet_manager.cc:1939] T aec54c64dce5482388d02b7dc0def7c2 P ae2d374cfe354e94b9d9ccc1fee3a5ad: Deleting consensus metadata
I20260927 16:39:20.484259 7918 catalog_manager.cc:4841] TS ae2d374cfe354e94b9d9ccc1fee3a5ad (127.0.98.3:33195): tablet aec54c64dce5482388d02b7dc0def7c2 (table test-table [id=97d30478c63c4a21b44dc04ffd8cae52]) successfully deleted
Sep 27 16:39:20 dist-test-slave-z40b krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 16:39:20 dist-test-slave-z40b krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:20 dist-test-slave-z40b krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 16:39:20.498404 8485 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:20.492319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34398 (local address 127.0.98.62:38577)
0927 16:39:20.492446 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:20.492451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:20.492696 (+ 245us) server_negotiation.cc:407] Connection header received
0927 16:39:20.492839 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:20.492843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:20.492893 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:20.492974 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:20.493652 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.494183 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.495007 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.495312 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.495434 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:20.498285 (+ 2851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34398: BlockingRecv error: recv got EOF from 127.0.0.1:34398 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 16:39:20 dist-test-slave-z40b krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 16:39:20.508502 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:20.500279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34412 (local address 127.0.98.62:38577)
0927 16:39:20.500388 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:20.500392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:20.500532 (+ 140us) server_negotiation.cc:407] Connection header received
0927 16:39:20.500972 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:20.500974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:20.501012 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:20.501105 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:20.501661 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.502142 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.502910 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.503114 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.503248 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:20.505441 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:20.505444 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:20.505446 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:20.505466 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:20.506856 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:20.507353 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:20.507357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:20.507359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:20.507438 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:20.507699 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:20.507703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:20.507704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:20.507862 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:20.507992 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:20.508213 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 16:39:20.508320 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 16:39:20.510993 7932 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:39:20.511459 7932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:39:20.515938 8071 tablet_service.cc:1467] Processing CreateTablet for tablet a531b4fbfceb4635916468af51f1280e (DEFAULT_TABLE table=test-table [id=7dd873c97e4c4bb59162f376c76251df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:39:20.516254 8071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a531b4fbfceb4635916468af51f1280e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:20.518707 8445 tablet_bootstrap.cc:492] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec: Bootstrap starting.
I20260927 16:39:20.519536 8445 tablet_bootstrap.cc:654] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:20.519479 8211 tablet_service.cc:1467] Processing CreateTablet for tablet a531b4fbfceb4635916468af51f1280e (DEFAULT_TABLE table=test-table [id=7dd873c97e4c4bb59162f376c76251df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:39:20.519739 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a531b4fbfceb4635916468af51f1280e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:20.521513 8446 tablet_bootstrap.cc:492] T a531b4fbfceb4635916468af51f1280e P 34362613bae544069a507d0f45be0de5: Bootstrap starting.
I20260927 16:39:20.521744 8351 tablet_service.cc:1467] Processing CreateTablet for tablet a531b4fbfceb4635916468af51f1280e (DEFAULT_TABLE table=test-table [id=7dd873c97e4c4bb59162f376c76251df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:39:20.522022 8351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a531b4fbfceb4635916468af51f1280e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:20.522404 8446 tablet_bootstrap.cc:654] T a531b4fbfceb4635916468af51f1280e P 34362613bae544069a507d0f45be0de5: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:20.523725 8446 tablet_bootstrap.cc:492] T a531b4fbfceb4635916468af51f1280e P 34362613bae544069a507d0f45be0de5: No bootstrap required, opened a new log
I20260927 16:39:20.523810 8446 ts_tablet_manager.cc:1397] T a531b4fbfceb4635916468af51f1280e P 34362613bae544069a507d0f45be0de5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 16:39:20.524255 8446 raft_consensus.cc:348] T a531b4fbfceb4635916468af51f1280e P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:20.524372 8446 raft_consensus.cc:374] T a531b4fbfceb4635916468af51f1280e P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:20.524420 8446 raft_consensus.cc:729] T a531b4fbfceb4635916468af51f1280e P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34362613bae544069a507d0f45be0de5, State: Initialized, Role: FOLLOWER
I20260927 16:39:20.524565 8446 consensus_queue.cc:260] T a531b4fbfceb4635916468af51f1280e P 34362613bae544069a507d0f45be0de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:20.524856 8446 ts_tablet_manager.cc:1428] T a531b4fbfceb4635916468af51f1280e P 34362613bae544069a507d0f45be0de5: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260927 16:39:20.524937 8447 tablet_bootstrap.cc:492] T a531b4fbfceb4635916468af51f1280e P ae2d374cfe354e94b9d9ccc1fee3a5ad: Bootstrap starting.
I20260927 16:39:20.525664 8447 tablet_bootstrap.cc:654] T a531b4fbfceb4635916468af51f1280e P ae2d374cfe354e94b9d9ccc1fee3a5ad: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:20.526688 8447 tablet_bootstrap.cc:492] T a531b4fbfceb4635916468af51f1280e P ae2d374cfe354e94b9d9ccc1fee3a5ad: No bootstrap required, opened a new log
I20260927 16:39:20.526767 8447 ts_tablet_manager.cc:1397] T a531b4fbfceb4635916468af51f1280e P ae2d374cfe354e94b9d9ccc1fee3a5ad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 16:39:20.527137 8445 tablet_bootstrap.cc:492] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec: No bootstrap required, opened a new log
I20260927 16:39:20.527105 8447 raft_consensus.cc:348] T a531b4fbfceb4635916468af51f1280e P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:20.527196 8447 raft_consensus.cc:374] T a531b4fbfceb4635916468af51f1280e P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:20.527263 8447 raft_consensus.cc:729] T a531b4fbfceb4635916468af51f1280e P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae2d374cfe354e94b9d9ccc1fee3a5ad, State: Initialized, Role: FOLLOWER
I20260927 16:39:20.527383 8445 ts_tablet_manager.cc:1397] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260927 16:39:20.527380 8447 consensus_queue.cc:260] T a531b4fbfceb4635916468af51f1280e P ae2d374cfe354e94b9d9ccc1fee3a5ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:20.527607 8447 ts_tablet_manager.cc:1428] T a531b4fbfceb4635916468af51f1280e P ae2d374cfe354e94b9d9ccc1fee3a5ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 16:39:20.527786 8445 raft_consensus.cc:348] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:20.527920 8445 raft_consensus.cc:374] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:20.527961 8445 raft_consensus.cc:729] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad11280495cd4070af56c658de93b7ec, State: Initialized, Role: FOLLOWER
I20260927 16:39:20.528103 8445 consensus_queue.cc:260] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:20.528368 8445 ts_tablet_manager.cc:1428] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 16:39:20.677634 8465 raft_consensus.cc:482] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:39:20.677767 8465 raft_consensus.cc:504] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:20.678208 8465 leader_election.cc:288] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34362613bae544069a507d0f45be0de5 (127.0.98.2:40193), ae2d374cfe354e94b9d9ccc1fee3a5ad (127.0.98.3:33195)
I20260927 16:39:20.678665 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a531b4fbfceb4635916468af51f1280e" candidate_uuid: "ad11280495cd4070af56c658de93b7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34362613bae544069a507d0f45be0de5" is_pre_election: true
I20260927 16:39:20.678714 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a531b4fbfceb4635916468af51f1280e" candidate_uuid: "ad11280495cd4070af56c658de93b7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" is_pre_election: true
I20260927 16:39:20.678823 8231 raft_consensus.cc:2454] T a531b4fbfceb4635916468af51f1280e P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad11280495cd4070af56c658de93b7ec in term 0.
I20260927 16:39:20.678865 8371 raft_consensus.cc:2454] T a531b4fbfceb4635916468af51f1280e P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad11280495cd4070af56c658de93b7ec in term 0.
I20260927 16:39:20.679142 8022 leader_election.cc:302] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34362613bae544069a507d0f45be0de5, ad11280495cd4070af56c658de93b7ec; no voters:
I20260927 16:39:20.679379 8465 raft_consensus.cc:2781] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:39:20.679435 8465 raft_consensus.cc:482] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:39:20.679461 8465 raft_consensus.cc:3037] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:20.680490 8465 raft_consensus.cc:504] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:20.680812 8465 leader_election.cc:288] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec [CANDIDATE]: Term 1 election: Requested vote from peers 34362613bae544069a507d0f45be0de5 (127.0.98.2:40193), ae2d374cfe354e94b9d9ccc1fee3a5ad (127.0.98.3:33195)
I20260927 16:39:20.681149 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a531b4fbfceb4635916468af51f1280e" candidate_uuid: "ad11280495cd4070af56c658de93b7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34362613bae544069a507d0f45be0de5"
I20260927 16:39:20.681188 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a531b4fbfceb4635916468af51f1280e" candidate_uuid: "ad11280495cd4070af56c658de93b7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad"
I20260927 16:39:20.681264 8231 raft_consensus.cc:3037] T a531b4fbfceb4635916468af51f1280e P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:20.681299 8371 raft_consensus.cc:3037] T a531b4fbfceb4635916468af51f1280e P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:20.682111 8371 raft_consensus.cc:2454] T a531b4fbfceb4635916468af51f1280e P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad11280495cd4070af56c658de93b7ec in term 1.
I20260927 16:39:20.682307 8231 raft_consensus.cc:2454] T a531b4fbfceb4635916468af51f1280e P 34362613bae544069a507d0f45be0de5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad11280495cd4070af56c658de93b7ec in term 1.
I20260927 16:39:20.682405 8022 leader_election.cc:302] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad11280495cd4070af56c658de93b7ec, ae2d374cfe354e94b9d9ccc1fee3a5ad; no voters:
I20260927 16:39:20.682565 8465 raft_consensus.cc:2781] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:39:20.682730 8465 raft_consensus.cc:686] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec [term 1 LEADER]: Becoming Leader. State: Replica: ad11280495cd4070af56c658de93b7ec, State: Running, Role: LEADER
I20260927 16:39:20.682893 8465 consensus_queue.cc:237] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:20.684562 7934 catalog_manager.cc:5495] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec reported cstate change: term changed from 0 to 1, leader changed from <none> to ad11280495cd4070af56c658de93b7ec (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ad11280495cd4070af56c658de93b7ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } health_report { overall_health: UNKNOWN } } }
I20260927 16:39:20.709148 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:20.706029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53304 (local address 127.0.98.1:36701)
0927 16:39:20.706147 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:20.706153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:20.706202 (+ 49us) server_negotiation.cc:407] Connection header received
0927 16:39:20.706366 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:20.706372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:20.706413 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:20.706498 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:39:20.706737 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.707267 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.708107 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.708319 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.708453 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:20.708711 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:39:20.708776 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:39:20.708856 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:20.708959 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 16:39:20.709011 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:39:20.710958 8371 raft_consensus.cc:1261] T a531b4fbfceb4635916468af51f1280e P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 1 FOLLOWER]: Refusing update from remote peer ad11280495cd4070af56c658de93b7ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:39:20.710958 8231 raft_consensus.cc:1261] T a531b4fbfceb4635916468af51f1280e P 34362613bae544069a507d0f45be0de5 [term 1 FOLLOWER]: Refusing update from remote peer ad11280495cd4070af56c658de93b7ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:39:20.711339 8465 consensus_queue.cc:1035] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:39:20.711485 8537 consensus_queue.cc:1035] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:39:20.719100 7934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:39:20.719261 7934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:39:20.720939 7934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ba0e51bba58480d8956cbc6689b1379: Sending DeleteTablet for 3 replicas of tablet a531b4fbfceb4635916468af51f1280e
I20260927 16:39:20.721388 8071 tablet_service.cc:1514] Processing DeleteTablet for tablet a531b4fbfceb4635916468af51f1280e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:39:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53298
I20260927 16:39:20.721469 8211 tablet_service.cc:1514] Processing DeleteTablet for tablet a531b4fbfceb4635916468af51f1280e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:39:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58440
I20260927 16:39:20.721493 8351 tablet_service.cc:1514] Processing DeleteTablet for tablet a531b4fbfceb4635916468af51f1280e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:39:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53644
I20260927 16:39:20.721558 8519 tablet_replica.cc:330] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec: stopping tablet replica
I20260927 16:39:20.721670 8520 tablet_replica.cc:330] T a531b4fbfceb4635916468af51f1280e P 34362613bae544069a507d0f45be0de5: stopping tablet replica
I20260927 16:39:20.721680 8519 raft_consensus.cc:2229] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:39:20.721781 8520 raft_consensus.cc:2229] T a531b4fbfceb4635916468af51f1280e P 34362613bae544069a507d0f45be0de5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:39:20.721845 8519 raft_consensus.cc:2258] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:39:20.721891 8520 raft_consensus.cc:2258] T a531b4fbfceb4635916468af51f1280e P 34362613bae544069a507d0f45be0de5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:39:20.722064 8521 tablet_replica.cc:330] T a531b4fbfceb4635916468af51f1280e P ae2d374cfe354e94b9d9ccc1fee3a5ad: stopping tablet replica
I20260927 16:39:20.722182 8521 raft_consensus.cc:2229] T a531b4fbfceb4635916468af51f1280e P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:39:20.722303 8521 raft_consensus.cc:2258] T a531b4fbfceb4635916468af51f1280e P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:39:20.722460 8519 ts_tablet_manager.cc:1905] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:39:20.722831 8520 ts_tablet_manager.cc:1905] T a531b4fbfceb4635916468af51f1280e P 34362613bae544069a507d0f45be0de5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:39:20.724440 8519 ts_tablet_manager.cc:1918] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:39:20.724514 8519 log.cc:1196] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-0/wal/wals/a531b4fbfceb4635916468af51f1280e
I20260927 16:39:20.724761 8520 ts_tablet_manager.cc:1918] T a531b4fbfceb4635916468af51f1280e P 34362613bae544069a507d0f45be0de5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:39:20.724766 8519 ts_tablet_manager.cc:1939] T a531b4fbfceb4635916468af51f1280e P ad11280495cd4070af56c658de93b7ec: Deleting consensus metadata
I20260927 16:39:20.724849 8520 log.cc:1196] T a531b4fbfceb4635916468af51f1280e P 34362613bae544069a507d0f45be0de5: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-1/wal/wals/a531b4fbfceb4635916468af51f1280e
I20260927 16:39:20.725129 8520 ts_tablet_manager.cc:1939] T a531b4fbfceb4635916468af51f1280e P 34362613bae544069a507d0f45be0de5: Deleting consensus metadata
I20260927 16:39:20.725972 7918 catalog_manager.cc:4841] TS 34362613bae544069a507d0f45be0de5 (127.0.98.2:40193): tablet a531b4fbfceb4635916468af51f1280e (table test-table [id=7dd873c97e4c4bb59162f376c76251df]) successfully deleted
I20260927 16:39:20.726119 7918 catalog_manager.cc:4841] TS ad11280495cd4070af56c658de93b7ec (127.0.98.1:36701): tablet a531b4fbfceb4635916468af51f1280e (table test-table [id=7dd873c97e4c4bb59162f376c76251df]) successfully deleted
I20260927 16:39:20.726282 8521 ts_tablet_manager.cc:1905] T a531b4fbfceb4635916468af51f1280e P ae2d374cfe354e94b9d9ccc1fee3a5ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:39:20.726258 7934 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:39:20.726634 7934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:39:20.728376 8521 ts_tablet_manager.cc:1918] T a531b4fbfceb4635916468af51f1280e P ae2d374cfe354e94b9d9ccc1fee3a5ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:39:20.728480 8521 log.cc:1196] T a531b4fbfceb4635916468af51f1280e P ae2d374cfe354e94b9d9ccc1fee3a5ad: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-2/wal/wals/a531b4fbfceb4635916468af51f1280e
I20260927 16:39:20.728746 8521 ts_tablet_manager.cc:1939] T a531b4fbfceb4635916468af51f1280e P ae2d374cfe354e94b9d9ccc1fee3a5ad: Deleting consensus metadata
I20260927 16:39:20.730427 7918 catalog_manager.cc:4841] TS ae2d374cfe354e94b9d9ccc1fee3a5ad (127.0.98.3:33195): tablet a531b4fbfceb4635916468af51f1280e (table test-table [id=7dd873c97e4c4bb59162f376c76251df]) successfully deleted
I20260927 16:39:20.730567 8071 tablet_service.cc:1467] Processing CreateTablet for tablet ba429818447b4cf9bb12e3e7321f6919 (DEFAULT_TABLE table=test-table [id=db8dfa916a384385b4aa217792e13ef2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:39:20.730679 8351 tablet_service.cc:1467] Processing CreateTablet for tablet ba429818447b4cf9bb12e3e7321f6919 (DEFAULT_TABLE table=test-table [id=db8dfa916a384385b4aa217792e13ef2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:39:20.730827 8071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba429818447b4cf9bb12e3e7321f6919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:20.730876 8351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba429818447b4cf9bb12e3e7321f6919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:20.730865 8211 tablet_service.cc:1467] Processing CreateTablet for tablet ba429818447b4cf9bb12e3e7321f6919 (DEFAULT_TABLE table=test-table [id=db8dfa916a384385b4aa217792e13ef2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:39:20.731057 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba429818447b4cf9bb12e3e7321f6919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:20.732728 8446 tablet_bootstrap.cc:492] T ba429818447b4cf9bb12e3e7321f6919 P 34362613bae544069a507d0f45be0de5: Bootstrap starting.
I20260927 16:39:20.732913 8445 tablet_bootstrap.cc:492] T ba429818447b4cf9bb12e3e7321f6919 P ad11280495cd4070af56c658de93b7ec: Bootstrap starting.
I20260927 16:39:20.732916 8447 tablet_bootstrap.cc:492] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad: Bootstrap starting.
I20260927 16:39:20.733538 8446 tablet_bootstrap.cc:654] T ba429818447b4cf9bb12e3e7321f6919 P 34362613bae544069a507d0f45be0de5: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:20.733659 8447 tablet_bootstrap.cc:654] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:20.733659 8445 tablet_bootstrap.cc:654] T ba429818447b4cf9bb12e3e7321f6919 P ad11280495cd4070af56c658de93b7ec: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:20.734691 8447 tablet_bootstrap.cc:492] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad: No bootstrap required, opened a new log
I20260927 16:39:20.734778 8447 ts_tablet_manager.cc:1397] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 16:39:20.734792 8446 tablet_bootstrap.cc:492] T ba429818447b4cf9bb12e3e7321f6919 P 34362613bae544069a507d0f45be0de5: No bootstrap required, opened a new log
I20260927 16:39:20.734877 8446 ts_tablet_manager.cc:1397] T ba429818447b4cf9bb12e3e7321f6919 P 34362613bae544069a507d0f45be0de5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 16:39:20.735008 8445 tablet_bootstrap.cc:492] T ba429818447b4cf9bb12e3e7321f6919 P ad11280495cd4070af56c658de93b7ec: No bootstrap required, opened a new log
I20260927 16:39:20.735085 8445 ts_tablet_manager.cc:1397] T ba429818447b4cf9bb12e3e7321f6919 P ad11280495cd4070af56c658de93b7ec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 16:39:20.735154 8447 raft_consensus.cc:348] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:20.735260 8447 raft_consensus.cc:374] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:20.735298 8447 raft_consensus.cc:729] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae2d374cfe354e94b9d9ccc1fee3a5ad, State: Initialized, Role: FOLLOWER
I20260927 16:39:20.735306 8446 raft_consensus.cc:348] T ba429818447b4cf9bb12e3e7321f6919 P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:20.735423 8446 raft_consensus.cc:374] T ba429818447b4cf9bb12e3e7321f6919 P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:20.735461 8446 raft_consensus.cc:729] T ba429818447b4cf9bb12e3e7321f6919 P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34362613bae544069a507d0f45be0de5, State: Initialized, Role: FOLLOWER
I20260927 16:39:20.735421 8447 consensus_queue.cc:260] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:20.735466 8445 raft_consensus.cc:348] T ba429818447b4cf9bb12e3e7321f6919 P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:20.735550 8445 raft_consensus.cc:374] T ba429818447b4cf9bb12e3e7321f6919 P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:20.735584 8445 raft_consensus.cc:729] T ba429818447b4cf9bb12e3e7321f6919 P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad11280495cd4070af56c658de93b7ec, State: Initialized, Role: FOLLOWER
I20260927 16:39:20.735562 8446 consensus_queue.cc:260] T ba429818447b4cf9bb12e3e7321f6919 P 34362613bae544069a507d0f45be0de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:20.735690 8447 ts_tablet_manager.cc:1428] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:39:20.735690 8445 consensus_queue.cc:260] T ba429818447b4cf9bb12e3e7321f6919 P ad11280495cd4070af56c658de93b7ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:20.735826 8446 ts_tablet_manager.cc:1428] T ba429818447b4cf9bb12e3e7321f6919 P 34362613bae544069a507d0f45be0de5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:39:20.735966 8445 ts_tablet_manager.cc:1428] T ba429818447b4cf9bb12e3e7321f6919 P ad11280495cd4070af56c658de93b7ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:39:20.826656 8505 raft_consensus.cc:482] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:39:20.826795 8505 raft_consensus.cc:504] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:20.827294 8505 leader_election.cc:288] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad11280495cd4070af56c658de93b7ec (127.0.98.1:36701), 34362613bae544069a507d0f45be0de5 (127.0.98.2:40193)
I20260927 16:39:20.827802 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba429818447b4cf9bb12e3e7321f6919" candidate_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad11280495cd4070af56c658de93b7ec" is_pre_election: true
I20260927 16:39:20.827987 8091 raft_consensus.cc:2454] T ba429818447b4cf9bb12e3e7321f6919 P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae2d374cfe354e94b9d9ccc1fee3a5ad in term 0.
I20260927 16:39:20.828297 8302 leader_election.cc:302] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad11280495cd4070af56c658de93b7ec, ae2d374cfe354e94b9d9ccc1fee3a5ad; no voters:
I20260927 16:39:20.828281 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba429818447b4cf9bb12e3e7321f6919" candidate_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34362613bae544069a507d0f45be0de5" is_pre_election: true
I20260927 16:39:20.828420 8231 raft_consensus.cc:2454] T ba429818447b4cf9bb12e3e7321f6919 P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae2d374cfe354e94b9d9ccc1fee3a5ad in term 0.
I20260927 16:39:20.828462 8505 raft_consensus.cc:2781] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:39:20.828536 8505 raft_consensus.cc:482] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:39:20.828596 8505 raft_consensus.cc:3037] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:20.829640 8505 raft_consensus.cc:504] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:20.829978 8505 leader_election.cc:288] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad [CANDIDATE]: Term 1 election: Requested vote from peers ad11280495cd4070af56c658de93b7ec (127.0.98.1:36701), 34362613bae544069a507d0f45be0de5 (127.0.98.2:40193)
I20260927 16:39:20.830305 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba429818447b4cf9bb12e3e7321f6919" candidate_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad11280495cd4070af56c658de93b7ec"
I20260927 16:39:20.830369 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba429818447b4cf9bb12e3e7321f6919" candidate_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34362613bae544069a507d0f45be0de5"
I20260927 16:39:20.830452 8091 raft_consensus.cc:3037] T ba429818447b4cf9bb12e3e7321f6919 P ad11280495cd4070af56c658de93b7ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:20.830474 8231 raft_consensus.cc:3037] T ba429818447b4cf9bb12e3e7321f6919 P 34362613bae544069a507d0f45be0de5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:20.831621 8091 raft_consensus.cc:2454] T ba429818447b4cf9bb12e3e7321f6919 P ad11280495cd4070af56c658de93b7ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae2d374cfe354e94b9d9ccc1fee3a5ad in term 1.
I20260927 16:39:20.831621 8231 raft_consensus.cc:2454] T ba429818447b4cf9bb12e3e7321f6919 P 34362613bae544069a507d0f45be0de5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae2d374cfe354e94b9d9ccc1fee3a5ad in term 1.
I20260927 16:39:20.831880 8302 leader_election.cc:302] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad11280495cd4070af56c658de93b7ec, ae2d374cfe354e94b9d9ccc1fee3a5ad; no voters:
I20260927 16:39:20.832042 8505 raft_consensus.cc:2781] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:39:20.832126 8505 raft_consensus.cc:686] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 1 LEADER]: Becoming Leader. State: Replica: ae2d374cfe354e94b9d9ccc1fee3a5ad, State: Running, Role: LEADER
I20260927 16:39:20.832275 8505 consensus_queue.cc:237] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } }
I20260927 16:39:20.833729 7932 catalog_manager.cc:5495] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad reported cstate change: term changed from 0 to 1, leader changed from <none> to ae2d374cfe354e94b9d9ccc1fee3a5ad (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae2d374cfe354e94b9d9ccc1fee3a5ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33195 } health_report { overall_health: HEALTHY } } }
I20260927 16:39:20.865832 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:20.862666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53658 (local address 127.0.98.3:33195)
0927 16:39:20.862948 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:20.862952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:20.862961 (+ 9us) server_negotiation.cc:407] Connection header received
0927 16:39:20.862993 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:20.862995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:20.863030 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:20.863083 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:39:20.863369 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.863840 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.864898 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.865088 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.865158 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:20.865384 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:39:20.865464 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:39:20.865522 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:20.865632 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 16:39:20.865671 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212}
I20260927 16:39:20.873324 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:20.870884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42083 (local address 127.0.98.3:33195)
0927 16:39:20.870986 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:20.870989 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:20.871000 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:39:20.871068 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:20.871070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:20.871102 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:20.871160 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:39:20.871393 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.871820 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.872538 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.872731 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.872797 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:20.873009 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:39:20.873070 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:39:20.873118 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:20.873190 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 16:39:20.873221 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 16:39:20.873335 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:20.870793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33195 (local address 127.0.98.3:42083)
0927 16:39:20.870927 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:20.870941 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:20.871009 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:20.871183 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:20.871185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:20.871190 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:39:20.871308 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:20.871312 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.871915 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.871917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:20.872448 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:20.872455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:20.872817 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:20.872819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:20.872880 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:20.872901 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:39:20.873136 (+ 235us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:39:20.873140 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 16:39:20.873184 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 16:39:20.873237 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 16:39:20.878336 8091 raft_consensus.cc:1261] T ba429818447b4cf9bb12e3e7321f6919 P ad11280495cd4070af56c658de93b7ec [term 1 FOLLOWER]: Refusing update from remote peer ae2d374cfe354e94b9d9ccc1fee3a5ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:39:20.878384 8231 raft_consensus.cc:1261] T ba429818447b4cf9bb12e3e7321f6919 P 34362613bae544069a507d0f45be0de5 [term 1 FOLLOWER]: Refusing update from remote peer ae2d374cfe354e94b9d9ccc1fee3a5ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:39:20.878686 8492 consensus_queue.cc:1035] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad11280495cd4070af56c658de93b7ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:39:20.878837 8505 consensus_queue.cc:1035] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "34362613bae544069a507d0f45be0de5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:39:20.907634 7932 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:39:20.907749 7932 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:39:20.909680 7932 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ba0e51bba58480d8956cbc6689b1379: Sending DeleteTablet for 3 replicas of tablet ba429818447b4cf9bb12e3e7321f6919
I20260927 16:39:20.910175 8071 tablet_service.cc:1514] Processing DeleteTablet for tablet ba429818447b4cf9bb12e3e7321f6919 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:39:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53298
I20260927 16:39:20.910284 8351 tablet_service.cc:1514] Processing DeleteTablet for tablet ba429818447b4cf9bb12e3e7321f6919 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:39:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53644
I20260927 16:39:20.910358 8519 tablet_replica.cc:330] T ba429818447b4cf9bb12e3e7321f6919 P ad11280495cd4070af56c658de93b7ec: stopping tablet replica
I20260927 16:39:20.910394 8211 tablet_service.cc:1514] Processing DeleteTablet for tablet ba429818447b4cf9bb12e3e7321f6919 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:39:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58440
I20260927 16:39:20.910444 8521 tablet_replica.cc:330] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad: stopping tablet replica
I20260927 16:39:20.910490 8519 raft_consensus.cc:2229] T ba429818447b4cf9bb12e3e7321f6919 P ad11280495cd4070af56c658de93b7ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:39:20.910542 8521 raft_consensus.cc:2229] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:39:20.910591 8519 raft_consensus.cc:2258] T ba429818447b4cf9bb12e3e7321f6919 P ad11280495cd4070af56c658de93b7ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:39:20.910801 8521 raft_consensus.cc:2258] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:39:20.911350 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 8006
I20260927 16:39:20.911451 8519 ts_tablet_manager.cc:1905] T ba429818447b4cf9bb12e3e7321f6919 P ad11280495cd4070af56c658de93b7ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:39:20.911537 8521 ts_tablet_manager.cc:1905] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:39:20.914330 8521 ts_tablet_manager.cc:1918] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:39:20.914402 8521 log.cc:1196] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-2/wal/wals/ba429818447b4cf9bb12e3e7321f6919
I20260927 16:39:20.914636 8521 ts_tablet_manager.cc:1939] T ba429818447b4cf9bb12e3e7321f6919 P ae2d374cfe354e94b9d9ccc1fee3a5ad: Deleting consensus metadata
I20260927 16:39:20.914928 8520 tablet_replica.cc:330] T ba429818447b4cf9bb12e3e7321f6919 P 34362613bae544069a507d0f45be0de5: stopping tablet replica
I20260927 16:39:20.915024 8520 raft_consensus.cc:2229] T ba429818447b4cf9bb12e3e7321f6919 P 34362613bae544069a507d0f45be0de5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:39:20.915143 8520 raft_consensus.cc:2258] T ba429818447b4cf9bb12e3e7321f6919 P 34362613bae544069a507d0f45be0de5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:39:20.915269 7918 catalog_manager.cc:4841] TS ae2d374cfe354e94b9d9ccc1fee3a5ad (127.0.98.3:33195): tablet ba429818447b4cf9bb12e3e7321f6919 (table test-table [id=db8dfa916a384385b4aa217792e13ef2]) successfully deleted
I20260927 16:39:20.915885 8520 ts_tablet_manager.cc:1905] T ba429818447b4cf9bb12e3e7321f6919 P 34362613bae544069a507d0f45be0de5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:39:20.918071 8520 ts_tablet_manager.cc:1918] T ba429818447b4cf9bb12e3e7321f6919 P 34362613bae544069a507d0f45be0de5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:39:20.918191 8520 log.cc:1196] T ba429818447b4cf9bb12e3e7321f6919 P 34362613bae544069a507d0f45be0de5: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790527121515808-392-0/minicluster-data/ts-1/wal/wals/ba429818447b4cf9bb12e3e7321f6919
I20260927 16:39:20.918491 8520 ts_tablet_manager.cc:1939] T ba429818447b4cf9bb12e3e7321f6919 P 34362613bae544069a507d0f45be0de5: Deleting consensus metadata
I20260927 16:39:20.919144 7918 catalog_manager.cc:4841] TS 34362613bae544069a507d0f45be0de5 (127.0.98.2:40193): tablet ba429818447b4cf9bb12e3e7321f6919 (table test-table [id=db8dfa916a384385b4aa217792e13ef2]) successfully deleted
W20260927 16:39:20.920423 7918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36701 (error 108)
I20260927 16:39:20.920717 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 8146
I20260927 16:39:20.922946 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:20.922556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36701 (local address 127.0.0.1:53314)
0927 16:39:20.922760 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:20.922837 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
W20260927 16:39:20.923072 7918 catalog_manager.cc:4584] TS ad11280495cd4070af56c658de93b7ec (127.0.98.1:36701): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ba429818447b4cf9bb12e3e7321f6919: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36701: connect: Connection refused (error 111)
I20260927 16:39:20.928284 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 8286
I20260927 16:39:20.934022 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 7902
2026-09-27T16:39:20Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3729 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:39:20 dist-test-slave-z40b krb5kdc[8555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:39:20 dist-test-slave-z40b krb5kdc[8555](info): set up 2 sockets
Sep 27 16:39:20 dist-test-slave-z40b krb5kdc[8555](info): commencing operation
krb5kdc: starting...
W20260927 16:39:21.955932 392 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:39:22 dist-test-slave-z40b krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:39:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:39:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:39:22.069634 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41433
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:22.170514 8568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:22.170763 8568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:22.170840 8568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:22.174609 8568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:39:22.174683 8568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:22.174719 8568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:22.174772 8568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:39:22.174808 8568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:39:22.179587 8568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:22.180445 8568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:22.181365 8568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:22.189131 8573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:22.189131 8574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:22.189208 8568 server_base.cc:1034] running on GCE node
W20260927 16:39:22.189172 8576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:22.189908 8568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:22.190902 8568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:22.192093 8568 hybrid_clock.cc:648] HybridClock initialized: now 1790527162192078 us; error 31 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T16:39:22Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1271 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:39:22 dist-test-slave-z40b krb5kdc[8581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:39:22 dist-test-slave-z40b krb5kdc[8581](info): set up 2 sockets
Sep 27 16:39:22 dist-test-slave-z40b krb5kdc[8581](info): commencing operation
krb5kdc: starting...
W20260927 16:39:23.219738 392 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:39:23 dist-test-slave-z40b krb5kdc[8581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:39:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:39:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:39:23.333408 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43195
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:23.435858 8594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:23.436076 8594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:23.436111 8594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:23.440014 8594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:39:23.440063 8594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:23.440083 8594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:23.440099 8594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:39:23.440114 8594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:39:23.445133 8594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:23.446060 8594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:23.446965 8594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:23.454907 8602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:23.455082 8594 server_base.cc:1034] running on GCE node
W20260927 16:39:23.454998 8600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:23.454906 8599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:23.455727 8594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:23.456753 8594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:23.457950 8594 hybrid_clock.cc:648] HybridClock initialized: now 1790527163457930 us; error 29 us; skew 500 ppm
Sep 27 16:39:23 dist-test-slave-z40b krb5kdc[8581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:23.460758 8594 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:39:23.461572 8594 webserver.cc:460] Webserver started at http://127.0.98.62:36327/ using document root <none> and password file <none>
I20260927 16:39:23.462117 8594 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:23.462199 8594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:23.462415 8594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:23.464370 8594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "842bbc42a9a34ac8a2d71982f9b815f9"
format_stamp: "Formatted at 2026-09-27 16:39:23 on dist-test-slave-z40b"
server_key: "073f25c2c608e901f6bac31d8e84fa27"
server_key_iv: "cb9bfa53d32afcbaf72b85087a66ce47"
server_key_version: "encryptionkey@0"
I20260927 16:39:23.464960 8594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "842bbc42a9a34ac8a2d71982f9b815f9"
format_stamp: "Formatted at 2026-09-27 16:39:23 on dist-test-slave-z40b"
server_key: "073f25c2c608e901f6bac31d8e84fa27"
server_key_iv: "cb9bfa53d32afcbaf72b85087a66ce47"
server_key_version: "encryptionkey@0"
I20260927 16:39:23.468462 8594 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:39:23.470682 8609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:23.471824 8594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:39:23.471958 8594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "842bbc42a9a34ac8a2d71982f9b815f9"
format_stamp: "Formatted at 2026-09-27 16:39:23 on dist-test-slave-z40b"
server_key: "073f25c2c608e901f6bac31d8e84fa27"
server_key_iv: "cb9bfa53d32afcbaf72b85087a66ce47"
server_key_version: "encryptionkey@0"
I20260927 16:39:23.472061 8594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:23.484999 8594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:23.488078 8594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:23.488281 8594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:23.494731 8594 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43195
I20260927 16:39:23.494788 8661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43195 every 8 connection(s)
I20260927 16:39:23.495621 8594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:39:23.498178 8662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:23.498638 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 8594
I20260927 16:39:23.498767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:39:23.499089 392 external_mini_cluster.cc:1428] Setting key 2d150fe8ec22c32bdc90e937a4aed00d
I20260927 16:39:23.504305 8662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9: Bootstrap starting.
I20260927 16:39:23.506628 8662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:23.507366 8662 log.cc:824] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9: Log is configured to *not* fsync() on all Append() calls
Sep 27 16:39:23 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:39:23.509326 8662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9: No bootstrap required, opened a new log
I20260927 16:39:23.512575 8662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842bbc42a9a34ac8a2d71982f9b815f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43195 } }
I20260927 16:39:23.512792 8662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:23.512878 8662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842bbc42a9a34ac8a2d71982f9b815f9, State: Initialized, Role: FOLLOWER
I20260927 16:39:23.513042 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:23.500385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39090 (local address 127.0.98.62:43195)
0927 16:39:23.500785 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:23.500806 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:23.500846 (+ 40us) server_negotiation.cc:407] Connection header received
0927 16:39:23.501661 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:23.501681 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:23.502056 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:23.502399 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:23.503018 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:23.504397 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:23.505207 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:23.505912 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:23.506060 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:23.508563 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:23.508571 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:23.508589 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:23.508618 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:23.510223 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:23.510650 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:23.510655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:23.510659 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:23.510757 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:23.511048 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:23.511053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:23.511055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:23.511476 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:23.511681 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:23.511953 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 16:39:23.512215 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":103,"threads_started":1}
I20260927 16:39:23.513366 8662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842bbc42a9a34ac8a2d71982f9b815f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43195 } }
I20260927 16:39:23.513501 8662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:39:23.513577 8662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:39:23.513664 8662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:23.514623 8662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842bbc42a9a34ac8a2d71982f9b815f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43195 } }
I20260927 16:39:23.514963 8662 leader_election.cc:302] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 842bbc42a9a34ac8a2d71982f9b815f9; no voters:
I20260927 16:39:23.515268 8662 leader_election.cc:288] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:39:23.515358 8667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:39:23.515560 8667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [term 1 LEADER]: Becoming Leader. State: Replica: 842bbc42a9a34ac8a2d71982f9b815f9, State: Running, Role: LEADER
I20260927 16:39:23.515950 8667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842bbc42a9a34ac8a2d71982f9b815f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43195 } }
I20260927 16:39:23.516477 8662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:39:23.517693 8669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "842bbc42a9a34ac8a2d71982f9b815f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842bbc42a9a34ac8a2d71982f9b815f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43195 } } }
I20260927 16:39:23.517802 8669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:23.518541 8668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 842bbc42a9a34ac8a2d71982f9b815f9. Latest consensus state: current_term: 1 leader_uuid: "842bbc42a9a34ac8a2d71982f9b815f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842bbc42a9a34ac8a2d71982f9b815f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43195 } } }
I20260927 16:39:23.518657 8668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:23.518662 8676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:39:23.521642 8676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:39:23.526777 8676 catalog_manager.cc:1347] Generated new cluster ID: a5a1fe67d52d47148f605cc3b18aa8c5
I20260927 16:39:23.526834 8676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:39:23.531955 8676 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:39:23.533046 8676 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:39:23.538470 8676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9: Generated new TSK 0
I20260927 16:39:23.539104 8676 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:39:23.565665 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43195
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:23.669073 8688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:23.669287 8688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:23.669324 8688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:23.673187 8688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:23.673233 8688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:23.673282 8688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:39:23.678223 8688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:23.679149 8688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:23.680132 8688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:23.688997 8693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:23.689113 8688 server_base.cc:1034] running on GCE node
W20260927 16:39:23.688997 8694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:23.689059 8696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:23.689698 8688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:23.690284 8688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:23.691444 8688 hybrid_clock.cc:648] HybridClock initialized: now 1790527163691431 us; error 22 us; skew 500 ppm
Sep 27 16:39:23 dist-test-slave-z40b krb5kdc[8581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:23.693997 8688 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:39:23.694793 8688 webserver.cc:460] Webserver started at http://127.0.98.1:41853/ using document root <none> and password file <none>
I20260927 16:39:23.695335 8688 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:23.695399 8688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:23.695557 8688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:23.697402 8688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/data/instance:
uuid: "7ce52173c561420fa0eae5a2b7206c4b"
format_stamp: "Formatted at 2026-09-27 16:39:23 on dist-test-slave-z40b"
server_key: "a3358390682f4eb70d7c44471566d6c6"
server_key_iv: "03c880ee5e689f3ca164302551ce5cd6"
server_key_version: "encryptionkey@0"
I20260927 16:39:23.697896 8688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7ce52173c561420fa0eae5a2b7206c4b"
format_stamp: "Formatted at 2026-09-27 16:39:23 on dist-test-slave-z40b"
server_key: "a3358390682f4eb70d7c44471566d6c6"
server_key_iv: "03c880ee5e689f3ca164302551ce5cd6"
server_key_version: "encryptionkey@0"
I20260927 16:39:23.701339 8688 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:39:23.703555 8703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:23.704694 8688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:39:23.704850 8688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "7ce52173c561420fa0eae5a2b7206c4b"
format_stamp: "Formatted at 2026-09-27 16:39:23 on dist-test-slave-z40b"
server_key: "a3358390682f4eb70d7c44471566d6c6"
server_key_iv: "03c880ee5e689f3ca164302551ce5cd6"
server_key_version: "encryptionkey@0"
I20260927 16:39:23.704957 8688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:23.719991 8688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:23.722579 8688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:23.722764 8688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:23.723408 8688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:23.724171 8688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:23.724242 8688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:23.724308 8688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:23.724361 8688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:23.732717 8688 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42221
I20260927 16:39:23.732756 8816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42221 every 8 connection(s)
I20260927 16:39:23.733532 8688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:39:23.740954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 8688
I20260927 16:39:23.741137 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:39:23.741437 392 external_mini_cluster.cc:1428] Setting key 891fa9ba4205649d27566e6d3f4cfcec
Sep 27 16:39:23 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:39:23.746764 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:23.735251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50875 (local address 127.0.98.62:43195)
0927 16:39:23.735379 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:23.735383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:23.736221 (+ 838us) server_negotiation.cc:407] Connection header received
0927 16:39:23.737495 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:23.737498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:23.737541 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:23.737610 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:23.738672 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:23.739140 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:23.740305 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:23.740523 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:23.740735 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:23.743060 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:23.743064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:23.743067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:23.743090 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:23.744716 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:23.745313 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:23.745316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:23.745317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:23.745386 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:23.745755 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:23.745757 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:23.745758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:23.745918 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:23.745997 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:23.746559 (+ 562us) server_negotiation.cc:299] Negotiation successful
0927 16:39:23.746651 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 16:39:23.747958 8819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:23.735637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.1:50875)
0927 16:39:23.736056 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:23.736111 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:23.737313 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:23.737764 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:23.737770 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:23.738113 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:23.738543 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:23.738554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:23.739302 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:23.739306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:23.740198 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:23.740207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:23.740640 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:23.740643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:23.740767 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:23.741366 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:23.741382 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:23.742912 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:23.744822 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:23.744828 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:23.744842 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:23.745209 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:23.745471 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:23.745475 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:23.745479 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:23.745661 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:23.746028 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:23.746036 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:23.746321 (+ 285us) client_negotiation.cc:769] Sending connection context
0927 16:39:23.746528 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 16:39:23.746854 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":97,"threads_started":1}
I20260927 16:39:23.749146 8817 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43195
I20260927 16:39:23.749370 8817 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:23.749812 8817 heartbeater.cc:511] Master 127.0.98.62:43195 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:39:23.751468 8626 ts_manager.cc:194] Registered new tserver with Master: 7ce52173c561420fa0eae5a2b7206c4b (127.0.98.1:42221)
I20260927 16:39:23.752614 8626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50875
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:39:23.769418 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43195
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:23.873225 8822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:23.873426 8822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:23.873486 8822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:23.877272 8822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:23.877318 8822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:23.877414 8822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:39:23.882274 8822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:23.883185 8822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:23.884115 8822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:23.892791 8828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:23.892838 8830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:23.892874 8827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:23.892956 8822 server_base.cc:1034] running on GCE node
I20260927 16:39:23.893381 8822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:23.894037 8822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:23.895183 8822 hybrid_clock.cc:648] HybridClock initialized: now 1790527163895153 us; error 46 us; skew 500 ppm
Sep 27 16:39:23 dist-test-slave-z40b krb5kdc[8581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:23.897748 8822 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:39:23.898533 8822 webserver.cc:460] Webserver started at http://127.0.98.2:41931/ using document root <none> and password file <none>
I20260927 16:39:23.899073 8822 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:23.899158 8822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:23.899392 8822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:23.901729 8822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/data/instance:
uuid: "4682e675c6364641b05642be04ab3c72"
format_stamp: "Formatted at 2026-09-27 16:39:23 on dist-test-slave-z40b"
server_key: "cc674d04e0106480fc7e38b190995518"
server_key_iv: "d0f228f3ce2cd04856f2fbfceda196c1"
server_key_version: "encryptionkey@0"
I20260927 16:39:23.902274 8822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4682e675c6364641b05642be04ab3c72"
format_stamp: "Formatted at 2026-09-27 16:39:23 on dist-test-slave-z40b"
server_key: "cc674d04e0106480fc7e38b190995518"
server_key_iv: "d0f228f3ce2cd04856f2fbfceda196c1"
server_key_version: "encryptionkey@0"
I20260927 16:39:23.905884 8822 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 16:39:23.908176 8837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:23.909341 8822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:39:23.909461 8822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "4682e675c6364641b05642be04ab3c72"
format_stamp: "Formatted at 2026-09-27 16:39:23 on dist-test-slave-z40b"
server_key: "cc674d04e0106480fc7e38b190995518"
server_key_iv: "d0f228f3ce2cd04856f2fbfceda196c1"
server_key_version: "encryptionkey@0"
I20260927 16:39:23.909551 8822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:23.923267 8822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:23.925838 8822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:23.926023 8822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:23.926558 8822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:23.927376 8822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:23.927421 8822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:23.927476 8822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:23.927507 8822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:23.935562 8822 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45529
I20260927 16:39:23.935586 8950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45529 every 8 connection(s)
I20260927 16:39:23.936339 8822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:39:23.944638 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 8822
I20260927 16:39:23.944749 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:39:23.945055 392 external_mini_cluster.cc:1428] Setting key e64d672eca3a4eaad654129bbab37f32
Sep 27 16:39:23 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:39:23.949482 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:23.937959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59883 (local address 127.0.98.62:43195)
0927 16:39:23.938080 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:23.938084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:23.938799 (+ 715us) server_negotiation.cc:407] Connection header received
0927 16:39:23.940225 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:23.940228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:23.940279 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:23.940382 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:23.941439 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:23.941953 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:23.942999 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:23.943198 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:23.943342 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:23.945646 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:23.945649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:23.945652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:23.945681 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:23.947341 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:23.947827 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:23.947833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:23.947837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:23.947923 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:23.948280 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:23.948287 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:23.948290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:23.948485 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:23.948558 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:23.949171 (+ 613us) server_negotiation.cc:299] Negotiation successful
0927 16:39:23.949297 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:39:23.950285 8953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:23.938283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.2:59883)
0927 16:39:23.938649 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:23.938684 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:23.940037 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:23.940527 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:23.940534 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:23.940863 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:23.941301 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:23.941312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:23.942069 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:23.942073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:23.942885 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:23.942893 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:23.943371 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:23.943376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:23.943522 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:23.944059 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:23.944072 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:23.945493 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:23.947482 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:23.947487 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:23.947497 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:23.947734 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:23.948014 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:23.948016 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:23.948018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:23.948171 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:23.948583 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:23.948592 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:23.948909 (+ 317us) client_negotiation.cc:769] Sending connection context
0927 16:39:23.949109 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 16:39:23.949439 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":92,"threads_started":1}
I20260927 16:39:23.951431 8951 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43195
I20260927 16:39:23.951668 8951 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:23.952222 8951 heartbeater.cc:511] Master 127.0.98.62:43195 requested a full tablet report, sending...
I20260927 16:39:23.953238 8626 ts_manager.cc:194] Registered new tserver with Master: 4682e675c6364641b05642be04ab3c72 (127.0.98.2:45529)
I20260927 16:39:23.953758 8626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59883
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:39:23.973515 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43195
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:24.115602 8956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:24.115981 8956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:24.116074 8956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:24.125671 8956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:24.125763 8956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:24.125929 8956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:39:24.137315 8956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:24.138703 8956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:24.140139 8956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:24.151450 8964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:24.151455 8961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:24.151469 8962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:24.151609 8956 server_base.cc:1034] running on GCE node
I20260927 16:39:24.152135 8956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:24.152717 8956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:24.153915 8956 hybrid_clock.cc:648] HybridClock initialized: now 1790527164153874 us; error 49 us; skew 500 ppm
Sep 27 16:39:24 dist-test-slave-z40b krb5kdc[8581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:24.156581 8956 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:39:24.157430 8956 webserver.cc:460] Webserver started at http://127.0.98.3:41559/ using document root <none> and password file <none>
I20260927 16:39:24.157933 8956 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:24.158017 8956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:24.158214 8956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:24.160182 8956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/data/instance:
uuid: "ee9f5132d7db45fa88f80c83139a3e15"
format_stamp: "Formatted at 2026-09-27 16:39:24 on dist-test-slave-z40b"
server_key: "1ca3cdf367f6faba81fb0a8137d569e1"
server_key_iv: "821beae3beeadcfad447e9c170dc3483"
server_key_version: "encryptionkey@0"
I20260927 16:39:24.160802 8956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ee9f5132d7db45fa88f80c83139a3e15"
format_stamp: "Formatted at 2026-09-27 16:39:24 on dist-test-slave-z40b"
server_key: "1ca3cdf367f6faba81fb0a8137d569e1"
server_key_iv: "821beae3beeadcfad447e9c170dc3483"
server_key_version: "encryptionkey@0"
I20260927 16:39:24.164363 8956 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:39:24.166693 8971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:24.167913 8956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:24.168049 8956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "ee9f5132d7db45fa88f80c83139a3e15"
format_stamp: "Formatted at 2026-09-27 16:39:24 on dist-test-slave-z40b"
server_key: "1ca3cdf367f6faba81fb0a8137d569e1"
server_key_iv: "821beae3beeadcfad447e9c170dc3483"
server_key_version: "encryptionkey@0"
I20260927 16:39:24.168148 8956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:24.191126 8956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:24.194072 8956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:24.194247 8956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:24.194815 8956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:24.195597 8956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:24.195662 8956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:24.195722 8956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:24.195763 8956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:24.205811 8956 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46143
I20260927 16:39:24.205847 9084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46143 every 8 connection(s)
I20260927 16:39:24.206624 8956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:39:24.209472 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 8956
I20260927 16:39:24.209585 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:39:24.209823 392 external_mini_cluster.cc:1428] Setting key 3689e7d94ddcd090abd120ab1dff43cb
Sep 27 16:39:24 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:39:24.220844 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:24.208295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42727 (local address 127.0.98.62:43195)
0927 16:39:24.208448 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:24.208453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:24.209351 (+ 898us) server_negotiation.cc:407] Connection header received
0927 16:39:24.211492 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:24.211495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:24.211548 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:24.211622 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:24.212876 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.213501 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.214566 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.214792 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.214896 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:24.217349 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:24.217352 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:24.217354 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:24.217390 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:24.218817 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:24.219298 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:24.219301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:24.219303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:24.219397 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:24.219716 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:24.219721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:24.219725 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:24.219983 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:24.220091 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:24.220476 (+ 385us) server_negotiation.cc:299] Negotiation successful
0927 16:39:24.220623 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 16:39:24.221334 9087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:24.208677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.3:42727)
0927 16:39:24.209166 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:24.209246 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:24.211204 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:24.211895 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:24.211902 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:24.212212 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:24.212722 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:24.212735 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.213624 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.213627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:24.214451 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:24.214459 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.214933 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.214937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:24.215089 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:24.215672 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:24.215686 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:24.217201 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:24.218925 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:24.218931 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:24.218942 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:24.219166 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:24.219497 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:24.219499 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:24.219500 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:24.219609 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:24.220083 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:24.220087 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:24.220265 (+ 178us) client_negotiation.cc:769] Sending connection context
0927 16:39:24.220389 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 16:39:24.220592 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":96,"threads_started":1}
I20260927 16:39:24.222492 9085 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43195
I20260927 16:39:24.222702 9085 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:24.223120 9085 heartbeater.cc:511] Master 127.0.98.62:43195 requested a full tablet report, sending...
I20260927 16:39:24.224135 8625 ts_manager.cc:194] Registered new tserver with Master: ee9f5132d7db45fa88f80c83139a3e15 (127.0.98.3:46143)
I20260927 16:39:24.224990 8625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42727
I20260927 16:39:24.235946 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 16:39:24.302551 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 8688
I20260927 16:39:24.307638 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 8822
I20260927 16:39:24.312671 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 8956
I20260927 16:39:24.318142 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 8594
I20260927 16:39:24.324151 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43195
--webserver_interface=127.0.98.62
--webserver_port=36327
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:24.424953 9094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:24.425149 9094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:24.425182 9094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:24.429194 9094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:39:24.429239 9094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:24.429257 9094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:24.429272 9094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:39:24.429319 9094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:39:24.434231 9094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:24.435127 9094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:24.436010 9094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:24.443568 9100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:24.443600 9102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:24.443619 9099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:24.443817 9094 server_base.cc:1034] running on GCE node
I20260927 16:39:24.444325 9094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:24.445206 9094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:24.446357 9094 hybrid_clock.cc:648] HybridClock initialized: now 1790527164446339 us; error 27 us; skew 500 ppm
Sep 27 16:39:24 dist-test-slave-z40b krb5kdc[8581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:24.448868 9094 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:39:24.449641 9094 webserver.cc:460] Webserver started at http://127.0.98.62:36327/ using document root <none> and password file <none>
I20260927 16:39:24.450076 9094 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:24.450136 9094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:24.453485 9094 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:39:24.455416 9109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:24.456521 9094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:24.456635 9094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "842bbc42a9a34ac8a2d71982f9b815f9"
format_stamp: "Formatted at 2026-09-27 16:39:23 on dist-test-slave-z40b"
server_key: "073f25c2c608e901f6bac31d8e84fa27"
server_key_iv: "cb9bfa53d32afcbaf72b85087a66ce47"
server_key_version: "encryptionkey@0"
I20260927 16:39:24.457083 9094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:24.473114 9094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:24.475582 9094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:24.475760 9094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:24.482570 9094 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43195
I20260927 16:39:24.482592 9161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43195 every 8 connection(s)
I20260927 16:39:24.483541 9094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:39:24.487229 9162 sys_catalog.cc:263] Verifying existing consensus state
I20260927 16:39:24.488745 9162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9: Bootstrap starting.
I20260927 16:39:24.489894 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 9094
I20260927 16:39:24.490372 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42221
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43195
--webserver_port=41853
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:39:24.505120 9162 log.cc:824] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:24.511441 9162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:39:24.511797 9162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9: Bootstrap complete.
I20260927 16:39:24.515969 9162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842bbc42a9a34ac8a2d71982f9b815f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43195 } }
I20260927 16:39:24.516350 9162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842bbc42a9a34ac8a2d71982f9b815f9, State: Initialized, Role: FOLLOWER
I20260927 16:39:24.516820 9162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842bbc42a9a34ac8a2d71982f9b815f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43195 } }
I20260927 16:39:24.516924 9162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:39:24.517014 9162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:39:24.517102 9162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [term 1 FOLLOWER]: Advancing to term 2
I20260927 16:39:24.518843 9162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842bbc42a9a34ac8a2d71982f9b815f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43195 } }
I20260927 16:39:24.519184 9162 leader_election.cc:302] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 842bbc42a9a34ac8a2d71982f9b815f9; no voters:
I20260927 16:39:24.519516 9162 leader_election.cc:288] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 16:39:24.519639 9166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 16:39:24.519852 9166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [term 2 LEADER]: Becoming Leader. State: Replica: 842bbc42a9a34ac8a2d71982f9b815f9, State: Running, Role: LEADER
I20260927 16:39:24.520126 9166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842bbc42a9a34ac8a2d71982f9b815f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43195 } }
I20260927 16:39:24.520643 9162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:39:24.521030 9168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 842bbc42a9a34ac8a2d71982f9b815f9. Latest consensus state: current_term: 2 leader_uuid: "842bbc42a9a34ac8a2d71982f9b815f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842bbc42a9a34ac8a2d71982f9b815f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43195 } } }
I20260927 16:39:24.521170 9168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:24.521399 9167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "842bbc42a9a34ac8a2d71982f9b815f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842bbc42a9a34ac8a2d71982f9b815f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43195 } } }
I20260927 16:39:24.521504 9167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9 [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:24.521799 9170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:39:24.524883 9170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:39:24.525477 9170 catalog_manager.cc:1259] Loaded cluster ID: a5a1fe67d52d47148f605cc3b18aa8c5
I20260927 16:39:24.525530 9170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:39:24.527828 9170 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:39:24.528430 9170 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 842bbc42a9a34ac8a2d71982f9b815f9: Loaded TSK: 0
I20260927 16:39:24.529592 9170 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 16:39:24.597208 9164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:24.597420 9164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:24.597460 9164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:24.601263 9164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:24.601308 9164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:24.601361 9164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:39:24.606177 9164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42221
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41853
--tserver_master_addrs=127.0.98.62:43195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:24.607049 9164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:24.607944 9164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:24.616776 9191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:24.616770 9188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:24.616779 9189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:24.617223 9164 server_base.cc:1034] running on GCE node
I20260927 16:39:24.617611 9164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:24.618208 9164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:24.619359 9164 hybrid_clock.cc:648] HybridClock initialized: now 1790527164619337 us; error 42 us; skew 500 ppm
Sep 27 16:39:24 dist-test-slave-z40b krb5kdc[8581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527164, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:24.622021 9164 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 16:39:24.622917 9164 webserver.cc:460] Webserver started at http://127.0.98.1:41853/ using document root <none> and password file <none>
I20260927 16:39:24.623507 9164 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:24.623595 9164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:24.627018 9164 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:39:24.628988 9198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:24.630115 9164 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 16:39:24.630196 9164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "7ce52173c561420fa0eae5a2b7206c4b"
format_stamp: "Formatted at 2026-09-27 16:39:23 on dist-test-slave-z40b"
server_key: "a3358390682f4eb70d7c44471566d6c6"
server_key_iv: "03c880ee5e689f3ca164302551ce5cd6"
server_key_version: "encryptionkey@0"
I20260927 16:39:24.630720 9164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:24.640637 9164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:24.643409 9164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:24.643575 9164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:24.644109 9164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:24.644809 9164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:24.644851 9164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:24.644917 9164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:24.644953 9164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:24.655157 9164 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42221
I20260927 16:39:24.655267 9311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42221 every 8 connection(s)
I20260927 16:39:24.656122 9164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 16:39:24 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:24.666167 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 9164
I20260927 16:39:24.666600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45529
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43195
--webserver_port=41931
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 16:39:24 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 16:39:24.668529 9314 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:24.658067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55801 (local address 127.0.98.62:43195)
0927 16:39:24.658515 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:24.658533 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:24.658815 (+ 282us) server_negotiation.cc:407] Connection header received
0927 16:39:24.660322 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:24.660331 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:24.660603 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:24.660931 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:24.662059 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.663012 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.664040 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.664479 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.664591 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:24.667872 (+ 3281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55801: BlockingRecv error: recv got EOF from 127.0.98.1:55801 (error 108)
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":111,"threads_started":1}
I20260927 16:39:24.668538 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:24.658155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.1:55801)
0927 16:39:24.658651 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:24.658692 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:24.659995 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:24.661040 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:24.661046 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:24.661444 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:24.661921 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:24.661934 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.663132 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.663134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:24.663943 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:24.663951 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.664604 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.664606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:24.664740 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:24.665308 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:24.665323 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:24.667695 (+ 2372us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":141,"threads_started":1}
W20260927 16:39:24.668905 9312 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43195 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43195: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 16:39:24 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:24 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 16:39:24.674068 9314 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:24.669396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34033 (local address 127.0.98.62:43195)
0927 16:39:24.669500 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:24.669504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:24.669516 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:39:24.669591 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:24.669594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:24.669630 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:24.669719 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:24.670053 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.670517 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.671335 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.671564 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.671671 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:24.673995 (+ 2324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34033: BlockingRecv error: recv got EOF from 127.0.98.1:34033 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:39:24.674069 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:24.669324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.1:34033)
0927 16:39:24.669422 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:24.669436 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:24.669507 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:24.669731 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:24.669733 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:24.669868 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:24.669950 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:24.669955 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.670617 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.670623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:24.671154 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:24.671162 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.671671 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.671674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:24.671766 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:24.672238 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:24.672260 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:24.673902 (+ 1642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
Sep 27 16:39:24 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:24 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:24.680251 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:24.674847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.1:50809)
0927 16:39:24.675026 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:24.675041 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:24.675115 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:24.675881 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:24.675883 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:24.675995 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:24.676054 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:24.676058 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.677103 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.677105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:24.677587 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:24.677593 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.678140 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.678142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:24.678192 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:24.678634 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:24.678644 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:24.680149 (+ 1505us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88}
W20260927 16:39:24.680410 9314 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:24.675362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50809 (local address 127.0.98.62:43195)
0927 16:39:24.675668 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:24.675675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:24.675690 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:39:24.675723 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:24.675727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:24.675773 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:24.675864 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:24.676163 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.677000 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.677688 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.678031 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.678137 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:24.680305 (+ 2168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50809: BlockingRecv error: recv got EOF from 127.0.98.1:50809 (error 108)
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":133,"threads_started":1}
W20260927 16:39:24.680430 9312 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 16:39:24.771741 9316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:24.771955 9316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:24.772018 9316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:24.775794 9316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:24.775836 9316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:24.775944 9316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:39:24.780779 9316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41931
--tserver_master_addrs=127.0.98.62:43195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:24.781654 9316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:24.782549 9316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:24.791165 9322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:24.791145 9323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:24.791145 9325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:24.791499 9316 server_base.cc:1034] running on GCE node
I20260927 16:39:24.791893 9316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:24.792471 9316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:24.793656 9316 hybrid_clock.cc:648] HybridClock initialized: now 1790527164793646 us; error 28 us; skew 500 ppm
Sep 27 16:39:24 dist-test-slave-z40b krb5kdc[8581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527164, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:24.796192 9316 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 16:39:24.796883 9316 webserver.cc:460] Webserver started at http://127.0.98.2:41931/ using document root <none> and password file <none>
I20260927 16:39:24.797360 9316 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:24.797420 9316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:24.800725 9316 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:39:24.802589 9332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:24.803730 9316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:24.803843 9316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "4682e675c6364641b05642be04ab3c72"
format_stamp: "Formatted at 2026-09-27 16:39:23 on dist-test-slave-z40b"
server_key: "cc674d04e0106480fc7e38b190995518"
server_key_iv: "d0f228f3ce2cd04856f2fbfceda196c1"
server_key_version: "encryptionkey@0"
I20260927 16:39:24.804307 9316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:24.836972 9316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:24.839713 9316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:24.839898 9316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:24.840504 9316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:24.841287 9316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:24.841331 9316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:24.841396 9316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:24.841436 9316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:24.849601 9316 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45529
I20260927 16:39:24.849622 9445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45529 every 8 connection(s)
I20260927 16:39:24.850502 9316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:39:24.852571 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 9316
I20260927 16:39:24.852979 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46143
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43195
--webserver_port=41559
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 16:39:24 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:24 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 16:39:24.863341 9314 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:24.852186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45107 (local address 127.0.98.62:43195)
0927 16:39:24.852324 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:24.852328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:24.853154 (+ 826us) server_negotiation.cc:407] Connection header received
0927 16:39:24.854848 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:24.854851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:24.854902 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:24.854982 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:24.856535 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.857045 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.858897 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.859104 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.859178 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:24.863171 (+ 3993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45107: BlockingRecv error: recv got EOF from 127.0.98.2:45107 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 16:39:24.864120 9448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:24.852554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.2:45107)
0927 16:39:24.853013 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:24.853060 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:24.854600 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:24.855161 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:24.855171 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:24.855724 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:24.856375 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:24.856397 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.857182 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.857189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:24.858776 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:24.858789 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.859288 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.859296 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:24.859527 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:24.860130 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:24.860155 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:24.863069 (+ 2914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":141,"threads_started":1}
W20260927 16:39:24.864562 9446 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43195 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43195: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 16:39:24 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:24 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:24.869789 9448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:24.864955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.2:43081)
0927 16:39:24.865067 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:24.865082 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:24.865148 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:24.865351 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:24.865353 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:24.865480 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:24.865560 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:24.865564 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.866539 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.866541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:24.867051 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:24.867057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.867613 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.867616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:24.867680 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:24.868124 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:24.868134 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:24.869712 (+ 1578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 16:39:24.869952 9314 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:24.865044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43081 (local address 127.0.98.62:43195)
0927 16:39:24.865139 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:24.865143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:24.865155 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:39:24.865211 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:24.865214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:24.865254 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:24.865332 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:24.865660 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.866441 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.867146 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.867506 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.867614 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:24.869846 (+ 2232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43081: BlockingRecv error: recv got EOF from 127.0.98.2:43081 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 16:39:24 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:24 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:24.876693 9450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:24.870207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.2:42367)
0927 16:39:24.870520 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:24.870533 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:24.870621 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:24.870807 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:24.870809 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:24.871022 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:24.871174 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:24.871184 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.871845 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.871848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:24.872746 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:24.872754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.873363 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.873368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:24.873459 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:24.874688 (+ 1229us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:24.874701 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:24.876541 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":83,"threads_started":1}
W20260927 16:39:24.876716 9314 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:24.870339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42367 (local address 127.0.98.62:43195)
0927 16:39:24.870446 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:24.870451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:24.870610 (+ 159us) server_negotiation.cc:407] Connection header received
0927 16:39:24.870688 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:24.870690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:24.870721 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:24.870781 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:24.871302 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.871745 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.872867 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:24.873173 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:24.873231 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:24.876648 (+ 3417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42367: BlockingRecv error: recv got EOF from 127.0.98.2:42367 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 16:39:24.876850 9446 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 16:39:24.962332 9449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:24.962549 9449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:24.962589 9449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:24.966300 9449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:24.966344 9449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:24.966398 9449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:39:24.971328 9449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46143
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41559
--tserver_master_addrs=127.0.98.62:43195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:24.972158 9449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:24.973042 9449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:24.981828 9458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:24.981840 9456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:24.981999 9449 server_base.cc:1034] running on GCE node
W20260927 16:39:24.981854 9455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:24.982517 9449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:24.983096 9449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:24.984269 9449 hybrid_clock.cc:648] HybridClock initialized: now 1790527164984239 us; error 41 us; skew 500 ppm
Sep 27 16:39:24 dist-test-slave-z40b krb5kdc[8581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527164, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:24.986872 9449 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 16:39:24.987713 9449 webserver.cc:460] Webserver started at http://127.0.98.3:41559/ using document root <none> and password file <none>
I20260927 16:39:24.988211 9449 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:24.988276 9449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:24.991794 9449 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:39:24.993628 9465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:24.994735 9449 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:24.994843 9449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "ee9f5132d7db45fa88f80c83139a3e15"
format_stamp: "Formatted at 2026-09-27 16:39:24 on dist-test-slave-z40b"
server_key: "1ca3cdf367f6faba81fb0a8137d569e1"
server_key_iv: "821beae3beeadcfad447e9c170dc3483"
server_key_version: "encryptionkey@0"
I20260927 16:39:24.995380 9449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:25.023646 9449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:25.026887 9449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:25.027082 9449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:25.027693 9449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:25.028609 9449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:25.028656 9449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:25.028723 9449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:25.028759 9449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:25.037652 9449 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46143
I20260927 16:39:25.037676 9578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46143 every 8 connection(s)
I20260927 16:39:25.038575 9449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:39:25.039342 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 9449
Sep 27 16:39:25 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:25.049006 9314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:25.040037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39092 (local address 127.0.98.62:43195)
0927 16:39:25.040172 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:25.040178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:25.040397 (+ 219us) server_negotiation.cc:407] Connection header received
0927 16:39:25.040566 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:25.040569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:25.040618 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:25.040693 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:25.041517 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:25.042036 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:25.042965 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:25.043309 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:25.043418 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:25.044943 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:25.044948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:25.044960 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:25.044983 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:25.046832 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:25.047320 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:25.047326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:25.047331 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:25.047479 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:25.047769 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:25.047772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:25.047773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:25.048172 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:25.048403 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:25.048622 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 16:39:25.048822 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 16:39:25 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 16:39:25.050331 9317 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:25.040729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33489 (local address 127.0.98.62:43195)
0927 16:39:25.040822 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:25.040833 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:25.041497 (+ 664us) server_negotiation.cc:407] Connection header received
0927 16:39:25.042822 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:25.042825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:25.042889 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:25.042979 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:25.044181 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:25.044974 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:25.046657 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:25.046867 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:25.047007 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:25.050237 (+ 3230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33489: BlockingRecv error: recv got EOF from 127.0.98.3:33489 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 16:39:25.051031 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:25.040915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.3:33489)
0927 16:39:25.041355 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:25.041402 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:25.042637 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:25.043150 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:25.043158 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:25.043602 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:25.044050 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:25.044062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:25.045085 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:25.045090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:25.046530 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:25.046542 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:25.047034 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:25.047037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:25.047182 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:25.047760 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:25.047780 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:25.050148 (+ 2368us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":126,"threads_started":1}
W20260927 16:39:25.051455 9579 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43195 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43195: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 16:39:25 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:25 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:25.056090 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:25.051800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.3:43285)
0927 16:39:25.051953 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:25.051968 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:25.052035 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:25.052222 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:25.052223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:25.052335 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:25.052410 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:25.052415 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:25.053036 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:25.053038 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:25.053541 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:25.053548 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:25.053907 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:25.053909 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:25.053973 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:25.054349 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:25.054359 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:25.056007 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":46}
W20260927 16:39:25.056221 9317 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:25.051879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43285 (local address 127.0.98.62:43195)
0927 16:39:25.051985 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:25.051988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:25.052009 (+ 21us) server_negotiation.cc:407] Connection header received
0927 16:39:25.052104 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:25.052106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:25.052136 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:25.052195 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:25.052507 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:25.052944 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:25.053630 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:25.053821 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:25.053890 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:25.056115 (+ 2225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43285: BlockingRecv error: recv got EOF from 127.0.98.3:43285 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 16:39:25 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:25 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:25.060822 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:25.056539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.3:57425)
0927 16:39:25.056633 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:25.056647 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:25.056716 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:25.056911 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:25.056914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:25.057058 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:25.057114 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:25.057118 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:25.057768 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:25.057771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:25.058281 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:25.058288 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:25.058693 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:25.058697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:25.058768 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:25.059113 (+ 345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:25.059122 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:25.060705 (+ 1583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 16:39:25.060889 9317 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:25.056618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57425 (local address 127.0.98.62:43195)
0927 16:39:25.056724 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:25.056728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:25.056741 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:39:25.056780 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:25.056784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:25.056822 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:25.056899 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:25.057214 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:25.057660 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:25.058387 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:25.058591 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:25.058692 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:25.060795 (+ 2103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57425: BlockingRecv error: recv got EOF from 127.0.98.3:57425 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 16:39:25.061066 9579 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 16:39:25 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:25 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:25.685799 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:25.680959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.1:38735)
0927 16:39:25.681248 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:25.681262 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:25.681340 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:25.681578 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:25.681580 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:25.681744 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:25.681842 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:25.681846 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:25.682556 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:25.682559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:25.683095 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:25.683102 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:25.683527 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:25.683530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:25.683598 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:25.684049 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:25.684059 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:25.685689 (+ 1630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":76,"threads_started":1}
W20260927 16:39:25.686005 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:25.681070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38735 (local address 127.0.98.62:43195)
0927 16:39:25.681315 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:25.681320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:25.681333 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:39:25.681426 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:25.681428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:25.681473 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:25.681545 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:25.681937 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:25.682470 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:25.683187 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:25.683426 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:25.683509 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:25.685865 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38735: BlockingRecv error: recv got EOF from 127.0.98.1:38735 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":99,"threads_started":1}
Sep 27 16:39:25 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:25 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:25.882905 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:25.877479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.2:60311)
0927 16:39:25.877752 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:25.877768 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:25.877872 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:25.878270 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:25.878274 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:25.878509 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:25.878663 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:25.878673 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:25.879625 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:25.879628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:25.880187 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:25.880194 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:25.880565 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:25.880568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:25.880637 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:25.881082 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:25.881092 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:25.882781 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":75,"threads_started":1}
W20260927 16:39:25.883023 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:25.877593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60311 (local address 127.0.98.62:43195)
0927 16:39:25.877733 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:25.877737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:25.877830 (+ 93us) server_negotiation.cc:407] Connection header received
0927 16:39:25.877996 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:25.877999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:25.878056 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:25.878129 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:25.878778 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:25.879516 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:25.880276 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:25.880476 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:25.880549 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:25.882902 (+ 2353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60311: BlockingRecv error: recv got EOF from 127.0.98.2:60311 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 16:39:26 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:26 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:26.067678 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:26.061612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.3:48743)
0927 16:39:26.061893 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:26.061906 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:26.062037 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:26.062301 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:26.062304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:26.062513 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:26.062617 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:26.062623 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:26.063480 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:26.063484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:26.064210 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:26.064231 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:26.064845 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:26.064848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:26.064965 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:26.065546 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:26.065566 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:26.067560 (+ 1994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":77,"threads_started":1}
W20260927 16:39:26.067806 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:26.061731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48743 (local address 127.0.98.62:43195)
0927 16:39:26.061865 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:26.061870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:26.061980 (+ 110us) server_negotiation.cc:407] Connection header received
0927 16:39:26.062141 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:26.062145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:26.062198 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:26.062281 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:26.062726 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:26.063354 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:26.064350 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:26.064689 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:26.064842 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:26.067692 (+ 2850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48743: BlockingRecv error: recv got EOF from 127.0.98.3:48743 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 16:39:26 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:26 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:26.691613 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:26.686489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.1:51109)
0927 16:39:26.686761 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:26.686775 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:26.686855 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:26.687115 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:26.687117 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:26.687300 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:26.687390 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:26.687395 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:26.688068 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:26.688070 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:26.688600 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:26.688607 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:26.688978 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:26.688980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:26.689035 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:26.689577 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:26.689589 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:26.691427 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":109,"threads_started":1}
W20260927 16:39:26.691670 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:26.686584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51109 (local address 127.0.98.62:43195)
0927 16:39:26.686801 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:26.686805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:26.686817 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:39:26.686938 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:26.686940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:26.686993 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:26.687067 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:26.687482 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:26.687982 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:26.688692 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:26.688893 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:26.688970 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:26.691550 (+ 2580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51109: BlockingRecv error: recv got EOF from 127.0.98.1:51109 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":78,"threads_started":1}
Sep 27 16:39:26 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:26 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:26.889070 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:26.883808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.2:45597)
0927 16:39:26.884126 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:26.884159 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:26.884272 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:26.884493 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:26.884495 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:26.884656 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:26.884777 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:26.884782 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:26.885501 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:26.885504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:26.886062 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:26.886069 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:26.886452 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:26.886454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:26.886512 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:26.887040 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:26.887057 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:26.888881 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":76,"threads_started":1}
W20260927 16:39:26.889104 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:26.883870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45597 (local address 127.0.98.62:43195)
0927 16:39:26.883996 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:26.884000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:26.884241 (+ 241us) server_negotiation.cc:407] Connection header received
0927 16:39:26.884358 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:26.884360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:26.884407 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:26.884474 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:26.884879 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:26.885410 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:26.886158 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:26.886362 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:26.886435 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:26.889006 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45597: BlockingRecv error: recv got EOF from 127.0.98.2:45597 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 16:39:27 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:27 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:27.073858 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:27.068448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.3:56067)
0927 16:39:27.068753 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:27.068769 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:27.068906 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:27.069197 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:27.069200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:27.069394 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:27.069481 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:27.069487 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:27.070247 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:27.070252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:27.070820 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:27.070827 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:27.071339 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:27.071343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:27.071427 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:27.071976 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:27.071988 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:27.073685 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":111,"threads_started":1}
W20260927 16:39:27.073904 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:27.068604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56067 (local address 127.0.98.62:43195)
0927 16:39:27.068753 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:27.068758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:27.068864 (+ 106us) server_negotiation.cc:407] Connection header received
0927 16:39:27.069019 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:27.069022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:27.069079 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:27.069173 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:27.069617 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:27.070142 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:27.070940 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:27.071170 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:27.071327 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:27.073803 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56067: BlockingRecv error: recv got EOF from 127.0.98.3:56067 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 16:39:27 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:27 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:27.703471 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:27.692447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.1:50219)
0927 16:39:27.692718 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:27.692731 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:27.692825 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:27.693052 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:27.693054 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:27.693222 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:27.693314 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:27.693319 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:27.694015 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:27.694017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:27.694569 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:27.694575 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:27.694964 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:27.694966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:27.695034 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:27.701279 (+ 6245us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:27.701304 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:27.703357 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":96,"threads_started":1}
W20260927 16:39:27.703644 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:27.692497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50219 (local address 127.0.98.62:43195)
0927 16:39:27.692718 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:27.692721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:27.692780 (+ 59us) server_negotiation.cc:407] Connection header received
0927 16:39:27.692887 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:27.692889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:27.692928 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:27.693008 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:27.693408 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:27.693916 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:27.694659 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:27.694858 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:27.694945 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:27.703481 (+ 8536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50219: BlockingRecv error: recv got EOF from 127.0.98.1:50219 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":79,"threads_started":1}
Sep 27 16:39:27 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:27 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:27.899887 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:27.889770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.2:37755)
0927 16:39:27.890028 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:27.890040 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:27.890134 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:27.890339 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:27.890341 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:27.890505 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:27.890597 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:27.890602 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:27.891311 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:27.891313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:27.891872 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:27.891879 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:27.892273 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:27.892275 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:27.892334 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:27.897916 (+ 5582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:27.897927 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:27.899779 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":105,"threads_started":1}
W20260927 16:39:27.900096 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:27.889844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37755 (local address 127.0.98.62:43195)
0927 16:39:27.890005 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:27.890009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:27.890109 (+ 100us) server_negotiation.cc:407] Connection header received
0927 16:39:27.890197 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:27.890199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:27.890240 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:27.890303 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:27.890689 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:27.891183 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:27.891964 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:27.892182 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:27.892256 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:27.899923 (+ 7667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37755: BlockingRecv error: recv got EOF from 127.0.98.2:37755 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 16:39:28 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:28 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:28.080946 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:28.074689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.3:37485)
0927 16:39:28.074934 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:28.074946 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:28.075071 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:28.075404 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:28.075407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:28.075572 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:28.075668 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:28.075673 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:28.076845 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:28.076847 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:28.077399 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:28.077409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:28.078094 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:28.078097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:28.078175 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:28.078720 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:28.078733 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:28.080785 (+ 2052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":83,"threads_started":1}
W20260927 16:39:28.081419 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:28.074709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37485 (local address 127.0.98.62:43195)
0927 16:39:28.074833 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:28.074838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:28.075036 (+ 198us) server_negotiation.cc:407] Connection header received
0927 16:39:28.075178 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:28.075183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:28.075284 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:28.075373 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:28.075785 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:28.076717 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:28.077514 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:28.077935 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:28.078074 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:28.080960 (+ 2886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37485: BlockingRecv error: recv got EOF from 127.0.98.3:37485 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 16:39:28 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:28 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:28.709762 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:28.704184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.1:36053)
0927 16:39:28.704429 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:28.704442 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:28.704535 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:28.704765 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:28.704767 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:28.704932 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:28.705023 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:28.705028 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:28.705705 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:28.705707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:28.706277 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:28.706285 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:28.706685 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:28.706687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:28.706746 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:28.707390 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:28.707407 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:28.709600 (+ 2193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":80,"threads_started":1}
W20260927 16:39:28.709825 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:28.704230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36053 (local address 127.0.98.62:43195)
0927 16:39:28.704468 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:28.704471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:28.704484 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:39:28.704617 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:28.704619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:28.704662 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:28.704734 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:28.705114 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:28.705620 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:28.706374 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:28.706593 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:28.706667 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:28.709713 (+ 3046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36053: BlockingRecv error: recv got EOF from 127.0.98.1:36053 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":73,"threads_started":1}
Sep 27 16:39:28 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:28 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:28.905997 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:28.900635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.2:46539)
0927 16:39:28.900913 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:28.900929 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:28.901058 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:28.901481 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:28.901485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:28.901713 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:28.901819 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:28.901824 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:28.902521 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:28.902523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:28.903143 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:28.903150 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:28.903604 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:28.903606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:28.903666 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:28.904148 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:28.904158 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:28.905864 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":86,"threads_started":1}
W20260927 16:39:28.906144 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:28.900696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46539 (local address 127.0.98.62:43195)
0927 16:39:28.900930 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:28.900934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:28.900990 (+ 56us) server_negotiation.cc:407] Connection header received
0927 16:39:28.901162 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:28.901165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:28.901237 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:28.901318 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:28.901917 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:28.902431 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:28.903288 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:28.903506 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:28.903588 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:28.905996 (+ 2408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46539: BlockingRecv error: recv got EOF from 127.0.98.2:46539 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 16:39:29 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:29 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:29.087090 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:29.081863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.3:58247)
0927 16:39:29.082130 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:29.082146 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:29.082260 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:29.082547 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:29.082550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:29.082789 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:29.082909 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:29.082916 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:29.083815 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:29.083818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:29.084356 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:29.084364 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:29.084751 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:29.084753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:29.084812 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:29.085236 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:29.085247 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:29.086970 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":83,"threads_started":1}
W20260927 16:39:29.087263 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:29.081888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58247 (local address 127.0.98.62:43195)
0927 16:39:29.082025 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:29.082042 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:29.082226 (+ 184us) server_negotiation.cc:407] Connection header received
0927 16:39:29.082352 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:29.082355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:29.082414 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:29.082504 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:29.083049 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:29.083722 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:29.084449 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:29.084662 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:29.084734 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:29.087092 (+ 2358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58247: BlockingRecv error: recv got EOF from 127.0.98.3:58247 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 16:39:29 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:29 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:29.715710 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:29.710581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.1:43323)
0927 16:39:29.710848 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:29.710862 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:29.710944 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:29.711169 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:29.711171 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:29.711359 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:29.711453 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:29.711458 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:29.712157 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:29.712159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:29.712733 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:29.712739 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:29.713134 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:29.713136 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:29.713190 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:29.713717 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:29.713729 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:29.715538 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":68,"threads_started":1}
W20260927 16:39:29.715775 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:29.710667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43323 (local address 127.0.98.62:43195)
0927 16:39:29.710913 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:29.710916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:29.710929 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:39:29.711022 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:29.711024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:29.711065 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:29.711133 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:29.711553 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:29.712069 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:29.712827 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:29.713047 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:29.713116 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:29.715655 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43323: BlockingRecv error: recv got EOF from 127.0.98.1:43323 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
Sep 27 16:39:29 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:29 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:29.912310 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:29.906729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.2:60273)
0927 16:39:29.907022 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:29.907038 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:29.907153 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:29.907453 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:29.907456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:29.907671 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:29.907757 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:29.907762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:29.908481 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:29.908486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:29.909006 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:29.909013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:29.909465 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:29.909468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:29.909559 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:29.910118 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:29.910136 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:29.912201 (+ 2065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":99,"threads_started":1}
W20260927 16:39:29.912516 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:29.906871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60273 (local address 127.0.98.62:43195)
0927 16:39:29.907015 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:29.907019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:29.907094 (+ 75us) server_negotiation.cc:407] Connection header received
0927 16:39:29.907292 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:29.907295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:29.907341 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:29.907431 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:29.907866 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:29.908379 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:29.909118 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:29.909343 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:29.909463 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:29.912330 (+ 2867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60273: BlockingRecv error: recv got EOF from 127.0.98.2:60273 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 16:39:30 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:30 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:30.094137 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:30.088073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.3:50205)
0927 16:39:30.088414 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:30.088427 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:30.088556 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:30.088811 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:30.088814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:30.089051 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:30.089207 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:30.089215 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:30.089995 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:30.089999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:30.090626 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:30.090634 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:30.091444 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:30.091449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:30.091586 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:30.092190 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:30.092205 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:30.094017 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":87,"threads_started":1}
W20260927 16:39:30.094291 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:30.088207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50205 (local address 127.0.98.62:43195)
0927 16:39:30.088329 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:30.088334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:30.088501 (+ 167us) server_negotiation.cc:407] Connection header received
0927 16:39:30.088648 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:30.088651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:30.088705 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:30.088789 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:30.089335 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:30.089883 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:30.090918 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:30.091334 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:30.091438 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:30.094139 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50205: BlockingRecv error: recv got EOF from 127.0.98.3:50205 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 16:39:30 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:30 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:30.721412 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:30.716519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.1:50727)
0927 16:39:30.716769 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:30.716782 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:30.716879 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:30.717123 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:30.717125 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:30.717290 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:30.717377 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:30.717382 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:30.718060 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:30.718062 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:30.718605 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:30.718612 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:30.718981 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:30.718983 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:30.719033 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:30.719576 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:30.719587 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:30.721305 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":80,"threads_started":1}
W20260927 16:39:30.721593 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:30.716559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50727 (local address 127.0.98.62:43195)
0927 16:39:30.716792 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:30.716795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:30.716848 (+ 53us) server_negotiation.cc:407] Connection header received
0927 16:39:30.716955 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:30.716957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:30.717019 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:30.717091 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:30.717466 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:30.717959 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:30.718695 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:30.718897 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:30.718965 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:30.721427 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50727: BlockingRecv error: recv got EOF from 127.0.98.1:50727 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":59,"threads_started":1}
Sep 27 16:39:30 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:30 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:30.918744 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:30.912987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.2:44577)
0927 16:39:30.913297 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:30.913310 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:30.913447 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:30.913740 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:30.913743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:30.913956 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:30.914042 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:30.914047 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:30.914779 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:30.914785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:30.915412 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:30.915421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:30.915852 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:30.915855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:30.915935 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:30.916527 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:30.916539 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:30.918628 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":67,"threads_started":1}
W20260927 16:39:30.918900 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:30.913061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44577 (local address 127.0.98.62:43195)
0927 16:39:30.913216 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:30.913221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:30.913388 (+ 167us) server_negotiation.cc:407] Connection header received
0927 16:39:30.913568 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:30.913573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:30.913626 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:30.913718 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:30.914154 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:30.914670 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:30.915529 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:30.915746 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:30.915849 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:30.918783 (+ 2934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44577: BlockingRecv error: recv got EOF from 127.0.98.2:44577 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 16:39:31 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:31 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:31.100505 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:31.094971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.3:33791)
0927 16:39:31.095305 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:31.095317 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:31.095445 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:31.095721 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:31.095723 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:31.095921 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:31.096005 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:31.096011 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:31.096771 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:31.096777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:31.097323 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:31.097331 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:31.097791 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:31.097794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:31.097870 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:31.098447 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:31.098459 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:31.100348 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":132,"threads_started":1}
W20260927 16:39:31.100593 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:31.095027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33791 (local address 127.0.98.62:43195)
0927 16:39:31.095195 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:31.095200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:31.095389 (+ 189us) server_negotiation.cc:407] Connection header received
0927 16:39:31.095574 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:31.095577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:31.095619 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:31.095701 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:31.096146 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:31.096665 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:31.097432 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:31.097679 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:31.097808 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:31.100472 (+ 2664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33791: BlockingRecv error: recv got EOF from 127.0.98.3:33791 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 16:39:31 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:31 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:31.727701 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:31.722301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.1:37021)
0927 16:39:31.722711 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:31.722729 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:31.722877 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:31.723276 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:31.723281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:31.723465 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:31.723561 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:31.723566 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:31.724297 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:31.724299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:31.724854 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:31.724860 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:31.725270 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:31.725272 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:31.725332 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:31.725788 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:31.725799 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:31.727593 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":81,"threads_started":1}
W20260927 16:39:31.727891 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:31.722349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37021 (local address 127.0.98.62:43195)
0927 16:39:31.723050 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:31.723053 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:31.723063 (+ 10us) server_negotiation.cc:407] Connection header received
0927 16:39:31.723092 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:31.723094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:31.723129 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:31.723194 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:31.723676 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:31.724185 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:31.724949 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:31.725177 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:31.725250 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:31.727751 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37021: BlockingRecv error: recv got EOF from 127.0.98.1:37021 (error 108)
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":75,"threads_started":1}
Sep 27 16:39:31 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:31 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:31.925666 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:31.919494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.2:56071)
0927 16:39:31.919840 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:31.919854 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:31.919989 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:31.920277 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:31.920280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:31.920557 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:31.920673 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:31.920680 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:31.921728 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:31.921733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:31.922649 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:31.922657 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:31.923219 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:31.923221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:31.923279 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:31.923738 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:31.923748 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:31.925533 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":98,"threads_started":1}
W20260927 16:39:31.925817 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:31.919664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56071 (local address 127.0.98.62:43195)
0927 16:39:31.919812 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:31.919816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:31.919933 (+ 117us) server_negotiation.cc:407] Connection header received
0927 16:39:31.920098 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:31.920101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:31.920159 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:31.920259 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:31.920792 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:31.921603 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:31.922803 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:31.923114 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:31.923189 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:31.925670 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56071: BlockingRecv error: recv got EOF from 127.0.98.2:56071 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 16:39:32 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:32 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:32.106602 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:32.101225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.3:39337)
0927 16:39:32.101569 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:32.101584 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:32.101682 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:32.101897 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:32.101899 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:32.102070 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:32.102174 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:32.102180 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:32.102974 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:32.102977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:32.103791 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:32.103800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:32.104197 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:32.104199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:32.104254 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:32.104697 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:32.104707 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:32.106481 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":137,"threads_started":1}
W20260927 16:39:32.106706 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:32.101290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39337 (local address 127.0.98.62:43195)
0927 16:39:32.101431 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:32.101435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:32.101652 (+ 217us) server_negotiation.cc:407] Connection header received
0927 16:39:32.101748 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:32.101750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:32.101794 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:32.101861 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:32.102284 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:32.102826 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:32.103890 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:32.104108 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:32.104181 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:32.106599 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39337: BlockingRecv error: recv got EOF from 127.0.98.3:39337 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 16:39:32 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:32 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:32.740985 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:32.728376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.1:47631)
0927 16:39:32.728642 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:32.728656 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:32.728744 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:32.729000 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:32.729002 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:32.729173 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:32.729265 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:32.729271 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:32.729966 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:32.729969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:32.730503 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:32.730509 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:32.730914 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:32.730916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:32.730972 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:32.739089 (+ 8117us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:32.739101 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:32.740876 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":95,"threads_started":1}
W20260927 16:39:32.741184 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:32.728516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47631 (local address 127.0.98.62:43195)
0927 16:39:32.728718 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:32.728721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:32.728734 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:39:32.728834 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:32.728837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:32.728891 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:32.728966 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:32.729361 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:32.729879 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:32.730600 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:32.730827 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:32.730897 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:32.741004 (+ 10107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47631: BlockingRecv error: recv got EOF from 127.0.98.1:47631 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":66,"threads_started":1}
Sep 27 16:39:32 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:32 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:32.932068 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:32.926360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.2:54195)
0927 16:39:32.926648 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:32.926660 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:32.926786 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:32.927057 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:32.927060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:32.927337 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:32.927428 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:32.927434 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:32.928301 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:32.928306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:32.928874 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:32.928882 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:32.929316 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:32.929319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:32.929399 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:32.929882 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:32.929892 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:32.931936 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":89,"threads_started":1}
W20260927 16:39:32.932170 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:32.926408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54195 (local address 127.0.98.62:43195)
0927 16:39:32.926570 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:32.926575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:32.926734 (+ 159us) server_negotiation.cc:407] Connection header received
0927 16:39:32.926889 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:32.926894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:32.926946 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:32.927032 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:32.927534 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:32.928193 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:32.928993 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:32.929211 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:32.929313 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:32.932063 (+ 2750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54195: BlockingRecv error: recv got EOF from 127.0.98.2:54195 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 16:39:33 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:33 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:33.112855 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:33.107305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.3:44919)
0927 16:39:33.107570 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:33.107609 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:33.107721 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:33.108023 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:33.108025 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:33.108190 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:33.108296 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:33.108301 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:33.108973 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:33.108976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:33.109489 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:33.109496 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:33.109875 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:33.109878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:33.109934 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:33.110399 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:33.110409 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:33.112691 (+ 2282us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":64,"threads_started":1}
W20260927 16:39:33.112910 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:33.107668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44919 (local address 127.0.98.62:43195)
0927 16:39:33.107841 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:33.107845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:33.107855 (+ 10us) server_negotiation.cc:407] Connection header received
0927 16:39:33.107885 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:33.107887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:33.107926 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:33.107986 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:33.108391 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:33.108889 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:33.109589 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:33.109787 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:33.109855 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:33.112805 (+ 2950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44919: BlockingRecv error: recv got EOF from 127.0.98.3:44919 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Sep 27 16:39:33 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:33 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:33.747008 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:33.741703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.1:46251)
0927 16:39:33.741993 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:33.742007 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:33.742102 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:33.742354 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:33.742356 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:33.742528 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:33.742629 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:33.742634 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:33.743350 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:33.743352 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:33.743903 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:33.743910 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:33.744307 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:33.744309 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:33.744367 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:33.744929 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:33.744941 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:33.746888 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":86,"threads_started":1}
W20260927 16:39:33.747207 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:33.741798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46251 (local address 127.0.98.62:43195)
0927 16:39:33.742025 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:33.742029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:33.742059 (+ 30us) server_negotiation.cc:407] Connection header received
0927 16:39:33.742200 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:33.742202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:33.742244 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:33.742318 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:33.742725 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:33.743266 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:33.744002 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:33.744218 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:33.744290 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:33.747012 (+ 2722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46251: BlockingRecv error: recv got EOF from 127.0.98.1:46251 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":68,"threads_started":1}
Sep 27 16:39:33 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:33 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:33.938233 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:33.932988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.2:60735)
0927 16:39:33.933276 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:33.933288 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:33.933381 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:33.933629 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:33.933631 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:33.933799 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:33.933891 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:33.933896 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:33.934691 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:33.934707 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:33.935283 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:33.935291 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:33.935697 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:33.935699 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:33.935759 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:33.936309 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:33.936320 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:33.938124 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":77,"threads_started":1}
W20260927 16:39:33.938447 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:33.933053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60735 (local address 127.0.98.62:43195)
0927 16:39:33.933219 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:33.933222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:33.933351 (+ 129us) server_negotiation.cc:407] Connection header received
0927 16:39:33.933441 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:33.933444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:33.933487 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:33.933568 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:33.933982 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:33.934565 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:33.935373 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:33.935605 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:33.935680 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:33.938227 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60735: BlockingRecv error: recv got EOF from 127.0.98.2:60735 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 16:39:34 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:34 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:34.119333 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:34.113797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.3:38265)
0927 16:39:34.114132 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:34.114144 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:34.114277 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:34.114564 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:34.114581 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:34.114787 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:34.114873 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:34.114879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:34.115683 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:34.115690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:34.116240 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:34.116248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:34.116694 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:34.116697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:34.116790 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:34.117316 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:34.117328 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:34.119115 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":104,"threads_started":1}
W20260927 16:39:34.119576 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:34.113852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38265 (local address 127.0.98.62:43195)
0927 16:39:34.114041 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:34.114046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:34.114218 (+ 172us) server_negotiation.cc:407] Connection header received
0927 16:39:34.114398 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:34.114403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:34.114455 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:34.114543 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:34.114984 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:34.115555 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:34.116365 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:34.116582 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:34.116684 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:34.119310 (+ 2626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38265: BlockingRecv error: recv got EOF from 127.0.98.3:38265 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 16:39:34 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:34 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:34.753211 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:34.747849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.1:48251)
0927 16:39:34.748088 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:34.748100 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:34.748181 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:34.748391 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:34.748393 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:34.748560 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:34.748660 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:34.748665 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:34.749347 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:34.749349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:34.750101 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:34.750118 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:34.750732 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:34.750735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:34.750793 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:34.751280 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:34.751302 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:34.753042 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":74,"threads_started":1}
W20260927 16:39:34.753285 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:34.747898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48251 (local address 127.0.98.62:43195)
0927 16:39:34.748144 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:34.748147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:34.748160 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:39:34.748245 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:34.748247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:34.748286 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:34.748359 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:34.748751 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:34.749261 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:34.750260 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:34.750623 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:34.750720 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:34.753165 (+ 2445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48251: BlockingRecv error: recv got EOF from 127.0.98.1:48251 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":67,"threads_started":1}
Sep 27 16:39:34 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:34 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:34.944796 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:34.939035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.2:50885)
0927 16:39:34.939352 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:34.939367 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:34.939495 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:34.939785 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:34.939788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:34.939993 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:34.940088 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:34.940093 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:34.940935 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:34.940939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:34.941852 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:34.941862 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:34.942307 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:34.942310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:34.942370 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:34.942837 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:34.942848 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:34.944689 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":88,"threads_started":1}
W20260927 16:39:34.944954 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:34.939099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50885 (local address 127.0.98.62:43195)
0927 16:39:34.939312 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:34.939317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:34.939451 (+ 134us) server_negotiation.cc:407] Connection header received
0927 16:39:34.939626 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:34.939628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:34.939679 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:34.939758 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:34.940203 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:34.940794 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:34.941962 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:34.942209 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:34.942289 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:34.944808 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50885: BlockingRecv error: recv got EOF from 127.0.98.2:50885 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Sep 27 16:39:35 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:35 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:35.125433 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:35.120130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.3:33317)
0927 16:39:35.120407 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:35.120424 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:35.120523 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:35.120789 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:35.120792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:35.121020 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:35.121144 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:35.121150 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:35.122018 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:35.122022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:35.122630 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:35.122638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:35.123114 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:35.123118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:35.123211 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:35.123665 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:35.123675 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:35.125325 (+ 1650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":92,"threads_started":1}
W20260927 16:39:35.125559 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:35.120228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33317 (local address 127.0.98.62:43195)
0927 16:39:35.120467 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:35.120470 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:35.120484 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:39:35.120623 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:35.120626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:35.120688 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:35.120767 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:35.121250 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:35.121898 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:35.122772 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:35.123006 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:35.123115 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:35.125448 (+ 2333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33317: BlockingRecv error: recv got EOF from 127.0.98.3:33317 (error 108)
Metrics: {"server-negotiator.queue_time_us":167}
Sep 27 16:39:35 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:35 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:35.758997 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:35.754040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.1:34013)
0927 16:39:35.754317 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:35.754331 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:35.754414 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:35.754652 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:35.754654 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:35.754824 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:35.754912 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:35.754916 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:35.755658 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:35.755660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:35.756206 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:35.756213 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:35.756614 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:35.756617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:35.756670 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:35.757146 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:35.757156 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:35.758891 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":79,"threads_started":1}
W20260927 16:39:35.759186 9624 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:35.754154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34013 (local address 127.0.98.62:43195)
0927 16:39:35.754366 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:35.754369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:35.754382 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:39:35.754494 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:35.754497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:35.754543 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:35.754618 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:35.755000 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:35.755546 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:35.756302 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:35.756525 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:35.756598 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:35.759016 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34013: BlockingRecv error: recv got EOF from 127.0.98.1:34013 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":51,"threads_started":1}
Sep 27 16:39:35 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:35 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:35.950515 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:35.945594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.2:34685)
0927 16:39:35.945865 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:35.945878 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:35.946005 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:35.946316 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:35.946319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:35.946495 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:35.946588 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:35.946593 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:35.947335 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:35.947341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:35.947895 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:35.947903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:35.948277 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:35.948279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:35.948337 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:35.948791 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:35.948802 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:35.950408 (+ 1606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":101,"threads_started":1}
W20260927 16:39:35.950752 9624 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:35.945628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34685 (local address 127.0.98.62:43195)
0927 16:39:35.945766 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:35.945772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:35.945974 (+ 202us) server_negotiation.cc:407] Connection header received
0927 16:39:35.946134 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:35.946138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:35.946192 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:35.946276 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:35.946696 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:35.947223 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:35.947988 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:35.948190 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:35.948259 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:35.950530 (+ 2271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34685: BlockingRecv error: recv got EOF from 127.0.98.2:34685 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 16:39:36 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:36 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:36.133003 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:36.126300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.3:48849)
0927 16:39:36.126604 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:36.126617 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:36.126751 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:36.127042 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:36.127045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:36.127321 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:36.127482 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:36.127491 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:36.128605 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:36.128608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:36.129240 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:36.129248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:36.129868 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:36.129872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:36.129965 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:36.130518 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:36.130567 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:36.132893 (+ 2326us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":64,"threads_started":1}
W20260927 16:39:36.133241 9624 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:36.126359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48849 (local address 127.0.98.62:43195)
0927 16:39:36.126490 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:36.126495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:36.126702 (+ 207us) server_negotiation.cc:407] Connection header received
0927 16:39:36.126842 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:36.126845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:36.126902 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:36.126986 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:36.127626 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:36.128493 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:36.129360 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:36.129756 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:36.129877 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:36.133066 (+ 3189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48849: BlockingRecv error: recv got EOF from 127.0.98.3:48849 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 16:39:36 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:36 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:36.764986 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:36.759733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.1:51201)
0927 16:39:36.759989 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:36.760003 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:36.760097 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:36.760347 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:36.760349 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:36.760525 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:36.760677 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:36.760689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:36.761457 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:36.761460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:36.761984 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:36.761990 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:36.762389 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:36.762391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:36.762449 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:36.763007 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:36.763018 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:36.764862 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":96,"threads_started":1}
W20260927 16:39:36.765161 9628 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:36.759801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51201 (local address 127.0.98.62:43195)
0927 16:39:36.760017 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:36.760020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:36.760052 (+ 32us) server_negotiation.cc:407] Connection header received
0927 16:39:36.760178 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:36.760180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:36.760222 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:36.760299 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:36.760787 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:36.761370 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:36.762074 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:36.762301 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:36.762371 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:36.764992 (+ 2621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51201: BlockingRecv error: recv got EOF from 127.0.98.1:51201 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":76,"threads_started":1}
Sep 27 16:39:36 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:36 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:36.956278 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:36.951228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.2:37079)
0927 16:39:36.951468 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:36.951480 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:36.951576 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:36.951789 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:36.951791 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:36.951955 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:36.952042 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:36.952047 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:36.952767 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:36.952769 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:36.953321 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:36.953329 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:36.953727 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:36.953730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:36.953786 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:36.954305 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:36.954316 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:36.956153 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":78,"threads_started":1}
W20260927 16:39:36.956383 9628 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:36.951258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37079 (local address 127.0.98.62:43195)
0927 16:39:36.951396 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:36.951401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:36.951549 (+ 148us) server_negotiation.cc:407] Connection header received
0927 16:39:36.951648 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:36.951650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:36.951693 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:36.951754 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:36.952148 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:36.952679 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:36.953426 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:36.953626 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:36.953711 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:36.956262 (+ 2551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37079: BlockingRecv error: recv got EOF from 127.0.98.2:37079 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 16:39:37 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:37 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:37.139001 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:37.133862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.3:39681)
0927 16:39:37.134148 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:37.134162 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:37.134269 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:37.134479 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:37.134481 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:37.134648 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:37.134744 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:37.134749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:37.135454 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:37.135457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:37.136001 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:37.136009 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:37.136448 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:37.136452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:37.136542 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:37.137096 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:37.137110 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:37.138893 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":102,"threads_started":1}
W20260927 16:39:37.139173 9628 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:37.133997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39681 (local address 127.0.98.62:43195)
0927 16:39:37.134148 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:37.134152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:37.134222 (+ 70us) server_negotiation.cc:407] Connection header received
0927 16:39:37.134343 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:37.134345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:37.134388 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:37.134444 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:37.134835 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:37.135368 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:37.136110 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:37.136333 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:37.136404 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:37.139005 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39681: BlockingRecv error: recv got EOF from 127.0.98.3:39681 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 16:39:37 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:37 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:37.777146 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:37.765763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.1:50767)
0927 16:39:37.766011 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:37.766023 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:37.766130 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:37.766393 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:37.766395 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:37.766565 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:37.766656 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:37.766662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:37.767387 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:37.767389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:37.767920 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:37.767926 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:37.768313 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:37.768315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:37.768370 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:37.775168 (+ 6798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:37.775179 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:37.776981 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":80,"threads_started":1}
W20260927 16:39:37.777216 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:37.765837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50767 (local address 127.0.98.62:43195)
0927 16:39:37.766050 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:37.766053 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:37.766075 (+ 22us) server_negotiation.cc:407] Connection header received
0927 16:39:37.766235 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:37.766237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:37.766279 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:37.766355 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:37.766751 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:37.767292 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:37.768010 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:37.768224 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:37.768295 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:37.777098 (+ 8803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50767: BlockingRecv error: recv got EOF from 127.0.98.1:50767 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":71,"threads_started":1}
Sep 27 16:39:37 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:37 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:37.962478 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:37.957087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.2:53895)
0927 16:39:37.957337 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:37.957350 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:37.957466 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:37.957717 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:37.957720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:37.957913 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:37.957999 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:37.958005 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:37.958726 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:37.958730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:37.959312 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:37.959320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:37.959733 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:37.959736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:37.959815 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:37.960334 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:37.960344 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:37.962368 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":88,"threads_started":1}
W20260927 16:39:37.962610 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:37.957322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53895 (local address 127.0.98.62:43195)
0927 16:39:37.957466 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:37.957470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:37.957483 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:39:37.957555 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:37.957559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:37.957604 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:37.957696 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:37.958114 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:37.958617 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:37.959418 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:37.959630 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:37.959729 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:37.962497 (+ 2768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53895: BlockingRecv error: recv got EOF from 127.0.98.2:53895 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 16:39:38 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:38 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:38.145112 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:38.139733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.3:60101)
0927 16:39:38.139975 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:38.139988 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:38.140081 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:38.140291 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:38.140292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:38.140462 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:38.140554 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:38.140559 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:38.141228 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:38.141231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:38.142055 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:38.142067 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:38.142576 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:38.142579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:38.142636 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:38.143168 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:38.143188 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:38.144951 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":83,"threads_started":1}
W20260927 16:39:38.145151 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:38.139774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60101 (local address 127.0.98.62:43195)
0927 16:39:38.139906 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:38.139909 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:38.140049 (+ 140us) server_negotiation.cc:407] Connection header received
0927 16:39:38.140149 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:38.140151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:38.140194 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:38.140260 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:38.140639 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:38.141132 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:38.142180 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:38.142479 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:38.142559 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:38.145058 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60101: BlockingRecv error: recv got EOF from 127.0.98.3:60101 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 16:39:38 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:38 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:38.782795 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:38.777905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.1:55843)
0927 16:39:38.778155 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:38.778167 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:38.778250 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:38.778454 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:38.778455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:38.778613 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:38.778716 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:38.778721 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:38.779515 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:38.779518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:38.780045 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:38.780053 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:38.780423 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:38.780425 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:38.780479 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:38.780939 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:38.780949 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:38.782615 (+ 1666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":94,"threads_started":1}
W20260927 16:39:38.782853 9636 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:38.777973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55843 (local address 127.0.98.62:43195)
0927 16:39:38.778244 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:38.778247 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:38.778259 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:39:38.778320 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:38.778322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:38.778360 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:38.778420 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:38.778805 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:38.779395 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:38.780138 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:38.780341 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:38.780410 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:38.782733 (+ 2323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55843: BlockingRecv error: recv got EOF from 127.0.98.1:55843 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":72,"threads_started":1}
Sep 27 16:39:38 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:38 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:38.968120 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:38.963188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.2:49101)
0927 16:39:38.963494 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:38.963508 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:38.963622 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:38.963872 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:38.963874 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:38.964055 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:38.964147 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:38.964152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:38.964868 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:38.964870 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:38.965415 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:38.965422 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:38.965798 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:38.965801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:38.965857 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:38.966346 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:38.966357 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:38.968017 (+ 1660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":81,"threads_started":1}
W20260927 16:39:38.968230 9636 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:38.963288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49101 (local address 127.0.98.62:43195)
0927 16:39:38.963424 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:38.963427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:38.963586 (+ 159us) server_negotiation.cc:407] Connection header received
0927 16:39:38.963709 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:38.963711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:38.963759 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:38.963826 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:38.964250 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:38.964782 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:38.965508 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:38.965705 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:38.965782 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:38.968127 (+ 2345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49101: BlockingRecv error: recv got EOF from 127.0.98.2:49101 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 16:39:39 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:39 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:39.151639 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:39.146002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.3:47081)
0927 16:39:39.146310 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:39.146323 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:39.146459 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:39.146730 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:39.146733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:39.146937 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:39.147025 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:39.147031 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:39.147827 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:39.147832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:39.148414 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:39.148422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:39.148866 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:39.148870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:39.148950 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:39.149442 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:39.149453 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:39.151512 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":95,"threads_started":1}
W20260927 16:39:39.151765 9636 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:39.146059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47081 (local address 127.0.98.62:43195)
0927 16:39:39.146249 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:39.146255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:39.146395 (+ 140us) server_negotiation.cc:407] Connection header received
0927 16:39:39.146558 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:39.146562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:39.146616 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:39.146703 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:39.147140 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:39.147718 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:39.148529 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:39.148759 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:39.148866 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:39.151628 (+ 2762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47081: BlockingRecv error: recv got EOF from 127.0.98.3:47081 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 16:39:39 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:39 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:39.788910 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:39.783745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.1:47365)
0927 16:39:39.784107 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:39.784123 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:39.784229 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:39.784515 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:39.784520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:39.784762 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:39.784859 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:39.784865 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:39.785555 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:39.785557 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:39.786087 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:39.786094 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:39.786516 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:39.786518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:39.786576 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:39.787037 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:39.787049 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:39.788793 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
W20260927 16:39:39.789076 9640 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:39.783801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47365 (local address 127.0.98.62:43195)
0927 16:39:39.784061 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:39.784064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:39.784207 (+ 143us) server_negotiation.cc:407] Connection header received
0927 16:39:39.784358 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:39.784360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:39.784404 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:39.784480 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:39.784960 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:39.785468 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:39.786208 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:39.786427 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:39.786500 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:39.788916 (+ 2416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47365: BlockingRecv error: recv got EOF from 127.0.98.1:47365 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
Sep 27 16:39:39 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:39:39 dist-test-slave-z40b krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:39:39.974048 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:39.969168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43195 (local address 127.0.98.2:40395)
0927 16:39:39.969446 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:39.969458 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:39.969543 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:39.969758 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:39.969760 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:39.969923 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:39.970009 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:39.970013 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:39.970673 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:39.970675 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:39.971197 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:39.971239 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:39.971592 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:39.971594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:39.971661 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:39.972134 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:39.972144 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:39.973917 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43195: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":104,"threads_started":1}
W20260927 16:39:39.974182 9640 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:39.969216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40395 (local address 127.0.98.62:43195)
0927 16:39:39.969362 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:39.969366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:39.969527 (+ 161us) server_negotiation.cc:407] Connection header received
0927 16:39:39.969615 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:39.969618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:39.969661 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:39.969725 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:39.970107 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:39.970588 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:39.971308 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:39.971505 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:39.971576 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:39.974069 (+ 2493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40395: BlockingRecv error: recv got EOF from 127.0.98.2:40395 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 16:39:40.042719 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 9164
I20260927 16:39:40.048120 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 9316
I20260927 16:39:40.053277 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 9449
I20260927 16:39:40.058149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 9094
2026-09-27T16:39:40Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17851 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T16:39:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:39:40Z Disabled control of system clock
I20260927 16:39:40.087080 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43561
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 16:39:40.193194 9645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:40.193526 9645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:40.193595 9645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:40.197974 9645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:39:40.198024 9645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:40.198038 9645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:40.198055 9645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:39:40.198105 9645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:39:40.202517 9645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43561
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:40.203454 9645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:40.204746 9645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:40.212389 9650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:40.212456 9651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:40.212396 9653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:40.212853 9645 server_base.cc:1034] running on GCE node
I20260927 16:39:40.213343 9645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:40.214421 9645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:40.215585 9645 hybrid_clock.cc:648] HybridClock initialized: now 1790527180215561 us; error 38 us; skew 500 ppm
I20260927 16:39:40.217173 9645 webserver.cc:460] Webserver started at http://127.0.98.62:35531/ using document root <none> and password file <none>
I20260927 16:39:40.217693 9645 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:40.217779 9645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:40.217984 9645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:40.219837 9645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "f807f067463c44a7a6d2f6c4ae697221"
format_stamp: "Formatted at 2026-09-27 16:39:40 on dist-test-slave-z40b"
server_key: "103694a4f27a021d54ae9b218909c703"
server_key_iv: "c37e2d1511684b489d17239c597f17b3"
server_key_version: "encryptionkey@0"
I20260927 16:39:40.220355 9645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "f807f067463c44a7a6d2f6c4ae697221"
format_stamp: "Formatted at 2026-09-27 16:39:40 on dist-test-slave-z40b"
server_key: "103694a4f27a021d54ae9b218909c703"
server_key_iv: "c37e2d1511684b489d17239c597f17b3"
server_key_version: "encryptionkey@0"
I20260927 16:39:40.223625 9645 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:39:40.225787 9659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:40.226881 9645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:40.227008 9645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "f807f067463c44a7a6d2f6c4ae697221"
format_stamp: "Formatted at 2026-09-27 16:39:40 on dist-test-slave-z40b"
server_key: "103694a4f27a021d54ae9b218909c703"
server_key_iv: "c37e2d1511684b489d17239c597f17b3"
server_key_version: "encryptionkey@0"
I20260927 16:39:40.227111 9645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:40.243598 9645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:40.244161 9645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:40.244338 9645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:40.251061 9645 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43561
I20260927 16:39:40.251075 9711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43561 every 8 connection(s)
I20260927 16:39:40.251929 9645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:39:40.254675 9712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:40.254944 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 9645
I20260927 16:39:40.255048 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:39:40.255345 392 external_mini_cluster.cc:1428] Setting key 3a1cbe8ed85028377e84b10ba323ed29
I20260927 16:39:40.260721 9712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f807f067463c44a7a6d2f6c4ae697221: Bootstrap starting.
I20260927 16:39:40.263104 9712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f807f067463c44a7a6d2f6c4ae697221: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:40.263979 9712 log.cc:824] T 00000000000000000000000000000000 P f807f067463c44a7a6d2f6c4ae697221: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:40.264976 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:40.256666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49800 (local address 127.0.98.62:43561)
0927 16:39:40.257327 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:40.257343 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:40.257384 (+ 41us) server_negotiation.cc:407] Connection header received
0927 16:39:40.258162 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:40.258184 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:40.258639 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:40.258982 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:40.259400 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:40.260727 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:40.261562 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:40.262212 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:40.262370 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:40.263107 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:40.263115 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:40.263133 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:39:40.263650 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:40.263713 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:39:40.263753 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:40.263847 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:40.263953 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 16:39:40.264186 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":94,"threads_started":1}
I20260927 16:39:40.265884 9712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f807f067463c44a7a6d2f6c4ae697221: No bootstrap required, opened a new log
I20260927 16:39:40.269181 9712 raft_consensus.cc:348] T 00000000000000000000000000000000 P f807f067463c44a7a6d2f6c4ae697221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f807f067463c44a7a6d2f6c4ae697221" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43561 } }
I20260927 16:39:40.269367 9712 raft_consensus.cc:374] T 00000000000000000000000000000000 P f807f067463c44a7a6d2f6c4ae697221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:40.269398 9712 raft_consensus.cc:729] T 00000000000000000000000000000000 P f807f067463c44a7a6d2f6c4ae697221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f807f067463c44a7a6d2f6c4ae697221, State: Initialized, Role: FOLLOWER
I20260927 16:39:40.269712 9712 consensus_queue.cc:260] T 00000000000000000000000000000000 P f807f067463c44a7a6d2f6c4ae697221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f807f067463c44a7a6d2f6c4ae697221" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43561 } }
I20260927 16:39:40.269816 9712 raft_consensus.cc:388] T 00000000000000000000000000000000 P f807f067463c44a7a6d2f6c4ae697221 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:39:40.269858 9712 raft_consensus.cc:482] T 00000000000000000000000000000000 P f807f067463c44a7a6d2f6c4ae697221 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:39:40.269920 9712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f807f067463c44a7a6d2f6c4ae697221 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:40.270596 9712 raft_consensus.cc:504] T 00000000000000000000000000000000 P f807f067463c44a7a6d2f6c4ae697221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f807f067463c44a7a6d2f6c4ae697221" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43561 } }
I20260927 16:39:40.270864 9712 leader_election.cc:302] T 00000000000000000000000000000000 P f807f067463c44a7a6d2f6c4ae697221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f807f067463c44a7a6d2f6c4ae697221; no voters:
I20260927 16:39:40.271099 9712 leader_election.cc:288] T 00000000000000000000000000000000 P f807f067463c44a7a6d2f6c4ae697221 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:39:40.271257 9717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f807f067463c44a7a6d2f6c4ae697221 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:39:40.271499 9717 raft_consensus.cc:686] T 00000000000000000000000000000000 P f807f067463c44a7a6d2f6c4ae697221 [term 1 LEADER]: Becoming Leader. State: Replica: f807f067463c44a7a6d2f6c4ae697221, State: Running, Role: LEADER
I20260927 16:39:40.271844 9717 consensus_queue.cc:237] T 00000000000000000000000000000000 P f807f067463c44a7a6d2f6c4ae697221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f807f067463c44a7a6d2f6c4ae697221" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43561 } }
I20260927 16:39:40.272286 9712 sys_catalog.cc:564] T 00000000000000000000000000000000 P f807f067463c44a7a6d2f6c4ae697221 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:39:40.273406 9719 sys_catalog.cc:455] T 00000000000000000000000000000000 P f807f067463c44a7a6d2f6c4ae697221 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f807f067463c44a7a6d2f6c4ae697221. Latest consensus state: current_term: 1 leader_uuid: "f807f067463c44a7a6d2f6c4ae697221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f807f067463c44a7a6d2f6c4ae697221" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43561 } } }
I20260927 16:39:40.273521 9719 sys_catalog.cc:458] T 00000000000000000000000000000000 P f807f067463c44a7a6d2f6c4ae697221 [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:40.273885 9718 sys_catalog.cc:455] T 00000000000000000000000000000000 P f807f067463c44a7a6d2f6c4ae697221 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f807f067463c44a7a6d2f6c4ae697221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f807f067463c44a7a6d2f6c4ae697221" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43561 } } }
I20260927 16:39:40.273942 9726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:39:40.274010 9718 sys_catalog.cc:458] T 00000000000000000000000000000000 P f807f067463c44a7a6d2f6c4ae697221 [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:40.276803 9726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:39:40.281405 9726 catalog_manager.cc:1347] Generated new cluster ID: bfc99524e0a843cd8ea41a420a5e8270
I20260927 16:39:40.281463 9726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:39:40.297243 9726 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:39:40.298033 9726 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:39:40.309928 9726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f807f067463c44a7a6d2f6c4ae697221: Generated new TSK 0
I20260927 16:39:40.310518 9726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:39:40.312999 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43561
--builtin_ntp_servers=127.0.98.20:43743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 16:39:40.415457 9736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:40.415664 9736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:40.415696 9736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:40.419641 9736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:40.419683 9736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:40.419764 9736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:39:40.424239 9736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:40.425069 9736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:40.426314 9736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:40.435289 9741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:40.435267 9742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:40.435276 9744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:40.435423 9736 server_base.cc:1034] running on GCE node
I20260927 16:39:40.435758 9736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:40.436278 9736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:40.437425 9736 hybrid_clock.cc:648] HybridClock initialized: now 1790527180437400 us; error 39 us; skew 500 ppm
I20260927 16:39:40.439038 9736 webserver.cc:460] Webserver started at http://127.0.98.1:36055/ using document root <none> and password file <none>
I20260927 16:39:40.439568 9736 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:40.439626 9736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:40.439776 9736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:40.441455 9736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/data/instance:
uuid: "3264b579e6214bd0b3d090d380b5e818"
format_stamp: "Formatted at 2026-09-27 16:39:40 on dist-test-slave-z40b"
server_key: "7a73b4fefdccab2ca3e1a26eed7ac5d8"
server_key_iv: "aa31e7315770f59108496b8c69f01035"
server_key_version: "encryptionkey@0"
I20260927 16:39:40.441910 9736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3264b579e6214bd0b3d090d380b5e818"
format_stamp: "Formatted at 2026-09-27 16:39:40 on dist-test-slave-z40b"
server_key: "7a73b4fefdccab2ca3e1a26eed7ac5d8"
server_key_iv: "aa31e7315770f59108496b8c69f01035"
server_key_version: "encryptionkey@0"
I20260927 16:39:40.445245 9736 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:39:40.447392 9750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:40.448566 9736 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:40.448648 9736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "3264b579e6214bd0b3d090d380b5e818"
format_stamp: "Formatted at 2026-09-27 16:39:40 on dist-test-slave-z40b"
server_key: "7a73b4fefdccab2ca3e1a26eed7ac5d8"
server_key_iv: "aa31e7315770f59108496b8c69f01035"
server_key_version: "encryptionkey@0"
I20260927 16:39:40.448734 9736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:40.470150 9736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:40.470608 9736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:40.470734 9736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:40.471155 9736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:40.471890 9736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:40.471931 9736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:40.471963 9736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:40.471982 9736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:40.479879 9736 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35127
I20260927 16:39:40.479902 9863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35127 every 8 connection(s)
I20260927 16:39:40.480631 9736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:39:40.488576 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 9736
I20260927 16:39:40.488692 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:39:40.488952 392 external_mini_cluster.cc:1428] Setting key 50599ed4d7e6810689cb8844c750eff2
I20260927 16:39:40.489034 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:40.482176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37401 (local address 127.0.98.62:43561)
0927 16:39:40.482303 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:40.482306 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:40.483234 (+ 928us) server_negotiation.cc:407] Connection header received
0927 16:39:40.484560 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:40.484563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:40.484619 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:40.484720 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:40.485531 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:40.486078 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:40.487179 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:40.487429 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:40.487539 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:40.488538 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:40.488541 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:40.488544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:39:40.488562 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:40.488579 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:39:40.488587 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:40.488661 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:40.488843 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 16:39:40.488898 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:39:40.489817 9866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:40.482497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43561 (local address 127.0.98.1:37401)
0927 16:39:40.483091 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:40.483126 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:40.484358 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:40.484874 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:40.484880 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:40.484954 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:40.485394 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:40.485405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:40.486226 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:40.486231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:40.487054 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:40.487063 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:40.487571 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:40.487575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:40.487723 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:40.488295 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:39:40.488296 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:40.488329 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:39:40.488382 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:40.488673 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:40.488684 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:40.488691 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 16:39:40.488819 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 16:39:40.489053 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":113,"threads_started":1}
I20260927 16:39:40.490934 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43561
--builtin_ntp_servers=127.0.98.20:43743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 16:39:40.490962 9864 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43561
I20260927 16:39:40.491173 9864 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:40.491612 9864 heartbeater.cc:511] Master 127.0.98.62:43561 requested a full tablet report, sending...
I20260927 16:39:40.493108 9676 ts_manager.cc:194] Registered new tserver with Master: 3264b579e6214bd0b3d090d380b5e818 (127.0.98.1:35127)
I20260927 16:39:40.494203 9676 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:37401
W20260927 16:39:40.593580 9867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:40.593806 9867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:40.593837 9867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:40.597570 9867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:40.597609 9867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:40.597652 9867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:39:40.601991 9867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:40.602804 9867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:40.604005 9867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:40.612671 9873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:40.612666 9875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:40.612668 9872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:40.612970 9867 server_base.cc:1034] running on GCE node
I20260927 16:39:40.613332 9867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:40.613914 9867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:40.615082 9867 hybrid_clock.cc:648] HybridClock initialized: now 1790527180615065 us; error 29 us; skew 500 ppm
I20260927 16:39:40.616744 9867 webserver.cc:460] Webserver started at http://127.0.98.2:42059/ using document root <none> and password file <none>
I20260927 16:39:40.617250 9867 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:40.617334 9867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:40.617522 9867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:40.619480 9867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/data/instance:
uuid: "239d08d198c84a1e9eb2557c346528c9"
format_stamp: "Formatted at 2026-09-27 16:39:40 on dist-test-slave-z40b"
server_key: "c3cfc050722542e79c76169609317a92"
server_key_iv: "81f4f30a9069eb52ce7915054d065550"
server_key_version: "encryptionkey@0"
I20260927 16:39:40.620075 9867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/wal/instance:
uuid: "239d08d198c84a1e9eb2557c346528c9"
format_stamp: "Formatted at 2026-09-27 16:39:40 on dist-test-slave-z40b"
server_key: "c3cfc050722542e79c76169609317a92"
server_key_iv: "81f4f30a9069eb52ce7915054d065550"
server_key_version: "encryptionkey@0"
I20260927 16:39:40.623477 9867 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:39:40.625651 9881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:40.626801 9867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:40.626925 9867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "239d08d198c84a1e9eb2557c346528c9"
format_stamp: "Formatted at 2026-09-27 16:39:40 on dist-test-slave-z40b"
server_key: "c3cfc050722542e79c76169609317a92"
server_key_iv: "81f4f30a9069eb52ce7915054d065550"
server_key_version: "encryptionkey@0"
I20260927 16:39:40.627025 9867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:40.648240 9867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:40.648811 9867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:40.648988 9867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:40.649480 9867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:40.650283 9867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:40.650349 9867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:40.650413 9867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:40.650457 9867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:40.659196 9867 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35931
I20260927 16:39:40.659374 9994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35931 every 8 connection(s)
I20260927 16:39:40.660074 9867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:39:40.668237 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 9867
I20260927 16:39:40.668346 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:39:40.668552 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:40.661762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40217 (local address 127.0.98.62:43561)
0927 16:39:40.661958 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:40.661963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:40.662988 (+ 1025us) server_negotiation.cc:407] Connection header received
0927 16:39:40.664310 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:40.664313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:40.664365 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:40.664452 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:40.665291 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:40.665754 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:40.666778 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:40.666979 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:40.667056 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:40.668037 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:40.668040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:40.668042 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:39:40.668053 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:40.668072 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:39:40.668080 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:40.668143 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:40.668355 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 16:39:40.668420 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 16:39:40.668680 392 external_mini_cluster.cc:1428] Setting key e9e5ea7a580f68cdb65c3cbc231b50b8
I20260927 16:39:40.669418 9997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:40.662228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43561 (local address 127.0.98.2:40217)
0927 16:39:40.662843 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:40.662879 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:40.664108 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:40.664589 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:40.664594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:40.664663 (+ 69us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:40.665056 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:40.665066 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:40.665852 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:40.665855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:40.666669 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:40.666678 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:40.667104 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:40.667107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:40.667279 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:40.667799 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:39:40.667801 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:40.667834 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:39:40.667886 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:40.668167 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:40.668175 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:40.668182 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 16:39:40.668319 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 16:39:40.668581 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":110,"threads_started":1}
I20260927 16:39:40.670367 9995 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43561
I20260927 16:39:40.670586 9995 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:40.670785 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43561
--builtin_ntp_servers=127.0.98.20:43743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 16:39:40.671150 9995 heartbeater.cc:511] Master 127.0.98.62:43561 requested a full tablet report, sending...
I20260927 16:39:40.672184 9676 ts_manager.cc:194] Registered new tserver with Master: 239d08d198c84a1e9eb2557c346528c9 (127.0.98.2:35931)
I20260927 16:39:40.672710 9676 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40217
W20260927 16:39:40.778970 9998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:40.779196 9998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:40.779309 9998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:40.783022 9998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:40.783063 9998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:40.783144 9998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:39:40.787820 9998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:40.788641 9998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:40.789831 9998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:40.798547 10004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:40.798646 9998 server_base.cc:1034] running on GCE node
W20260927 16:39:40.798600 10006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:40.798550 10003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:40.799136 9998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:40.799747 9998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:40.800900 9998 hybrid_clock.cc:648] HybridClock initialized: now 1790527180800875 us; error 36 us; skew 500 ppm
I20260927 16:39:40.802570 9998 webserver.cc:460] Webserver started at http://127.0.98.3:45757/ using document root <none> and password file <none>
I20260927 16:39:40.803045 9998 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:40.803098 9998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:40.803313 9998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:40.805136 9998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/data/instance:
uuid: "32acc9ab63e047419cd446aafc7b9156"
format_stamp: "Formatted at 2026-09-27 16:39:40 on dist-test-slave-z40b"
server_key: "eb11f6e446511bb37b4d1f6a91a7cf63"
server_key_iv: "305ccf8fced2c4bb1dbf614db7aaea7b"
server_key_version: "encryptionkey@0"
I20260927 16:39:40.805605 9998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/wal/instance:
uuid: "32acc9ab63e047419cd446aafc7b9156"
format_stamp: "Formatted at 2026-09-27 16:39:40 on dist-test-slave-z40b"
server_key: "eb11f6e446511bb37b4d1f6a91a7cf63"
server_key_iv: "305ccf8fced2c4bb1dbf614db7aaea7b"
server_key_version: "encryptionkey@0"
I20260927 16:39:40.809064 9998 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:39:40.811389 10012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:40.812638 9998 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:39:40.812767 9998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "32acc9ab63e047419cd446aafc7b9156"
format_stamp: "Formatted at 2026-09-27 16:39:40 on dist-test-slave-z40b"
server_key: "eb11f6e446511bb37b4d1f6a91a7cf63"
server_key_iv: "305ccf8fced2c4bb1dbf614db7aaea7b"
server_key_version: "encryptionkey@0"
I20260927 16:39:40.812875 9998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:40.837570 9998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:40.838166 9998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:40.838325 9998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:40.838822 9998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:40.839767 9998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:40.839814 9998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:40.839852 9998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:40.839872 9998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:40.848199 9998 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34459
I20260927 16:39:40.848227 10125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34459 every 8 connection(s)
I20260927 16:39:40.849136 9998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:39:40.857661 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:40.850765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49855 (local address 127.0.98.62:43561)
0927 16:39:40.850920 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:40.850925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:40.852081 (+ 1156us) server_negotiation.cc:407] Connection header received
0927 16:39:40.853493 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:40.853495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:40.853539 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:40.853615 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:40.854400 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:40.854926 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:40.856037 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:40.856245 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:40.856327 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:40.857216 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:40.857219 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:40.857221 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:39:40.857232 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:40.857249 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:39:40.857256 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:40.857322 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:40.857447 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 16:39:40.857492 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 16:39:40.857815 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 9998
I20260927 16:39:40.857930 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:39:40.858219 392 external_mini_cluster.cc:1428] Setting key c13bdcce6c7b319951673540bb8de549
I20260927 16:39:40.858495 10128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:40.851182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43561 (local address 127.0.98.3:49855)
0927 16:39:40.851924 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:40.851960 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:40.853316 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:40.853780 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:40.853786 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:40.853861 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:40.854287 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:40.854298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:40.855031 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:40.855033 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:40.855939 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:40.855947 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:40.856368 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:40.856370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:40.856503 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:40.856991 (+ 488us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:39:40.856992 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:40.857027 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:39:40.857088 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:40.857336 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:40.857342 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:40.857345 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 16:39:40.857430 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 16:39:40.857613 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":102,"threads_started":1}
I20260927 16:39:40.859614 10126 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43561
I20260927 16:39:40.859851 10126 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:40.860400 10126 heartbeater.cc:511] Master 127.0.98.62:43561 requested a full tablet report, sending...
I20260927 16:39:40.861294 9675 ts_manager.cc:194] Registered new tserver with Master: 32acc9ab63e047419cd446aafc7b9156 (127.0.98.3:34459)
I20260927 16:39:40.861835 9675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:49855
I20260927 16:39:40.871907 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 16:39:40.874493 9715 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:40.873778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49814 (local address 127.0.98.62:43561)
0927 16:39:40.873860 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:40.873863 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:40.874042 (+ 179us) server_negotiation.cc:407] Connection header received
0927 16:39:40.874109 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:40.874111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:40.874141 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:40.874217 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:40.874377 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49814: BlockingRecv error: recv got EOF from 127.0.0.1:49814 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 16:39:40.874904 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 9736
I20260927 16:39:40.880605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 9867
I20260927 16:39:40.885267 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 9998
I20260927 16:39:40.890213 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 9645
2026-09-27T16:39:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (830 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T16:39:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:39:40Z Disabled control of system clock
I20260927 16:39:40.917666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33775
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 16:39:41.020464 10137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:41.020684 10137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:41.020717 10137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:41.024706 10137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:39:41.024752 10137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:41.024767 10137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:41.024788 10137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:39:41.024802 10137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:39:41.029196 10137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33775
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:41.030124 10137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:41.031396 10137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:41.039017 10142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:41.039028 10145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:41.039027 10143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:41.039353 10137 server_base.cc:1034] running on GCE node
I20260927 16:39:41.039878 10137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:41.040879 10137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:41.042033 10137 hybrid_clock.cc:648] HybridClock initialized: now 1790527181042017 us; error 28 us; skew 500 ppm
I20260927 16:39:41.043655 10137 webserver.cc:460] Webserver started at http://127.0.98.62:42911/ using document root <none> and password file <none>
I20260927 16:39:41.044133 10137 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:41.044188 10137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:41.044344 10137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:41.046105 10137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "976f70b6a7f24f18bc380e0a5555b88c"
format_stamp: "Formatted at 2026-09-27 16:39:41 on dist-test-slave-z40b"
server_key: "d8cd809e4d56a822851b7a69db672a53"
server_key_iv: "da20b93d8b869bf8c10908437f88e92c"
server_key_version: "encryptionkey@0"
I20260927 16:39:41.046617 10137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "976f70b6a7f24f18bc380e0a5555b88c"
format_stamp: "Formatted at 2026-09-27 16:39:41 on dist-test-slave-z40b"
server_key: "d8cd809e4d56a822851b7a69db672a53"
server_key_iv: "da20b93d8b869bf8c10908437f88e92c"
server_key_version: "encryptionkey@0"
I20260927 16:39:41.049830 10137 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:39:41.052026 10151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:41.053138 10137 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:41.053228 10137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "976f70b6a7f24f18bc380e0a5555b88c"
format_stamp: "Formatted at 2026-09-27 16:39:41 on dist-test-slave-z40b"
server_key: "d8cd809e4d56a822851b7a69db672a53"
server_key_iv: "da20b93d8b869bf8c10908437f88e92c"
server_key_version: "encryptionkey@0"
I20260927 16:39:41.053313 10137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:41.057125 10137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:41.057575 10137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:41.057689 10137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:41.063371 10137 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33775
I20260927 16:39:41.063380 10203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33775 every 8 connection(s)
I20260927 16:39:41.064177 10137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:39:41.067030 10204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:41.071890 10204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 976f70b6a7f24f18bc380e0a5555b88c: Bootstrap starting.
I20260927 16:39:41.072256 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 10137
I20260927 16:39:41.072371 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:39:41.072613 392 external_mini_cluster.cc:1428] Setting key f2e7aab4677c8208af315043f14d0079
I20260927 16:39:41.074331 10204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 976f70b6a7f24f18bc380e0a5555b88c: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:41.074971 10204 log.cc:824] T 00000000000000000000000000000000 P 976f70b6a7f24f18bc380e0a5555b88c: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:41.076854 10204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 976f70b6a7f24f18bc380e0a5555b88c: No bootstrap required, opened a new log
I20260927 16:39:41.077694 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:41.073861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53570 (local address 127.0.98.62:33775)
0927 16:39:41.074434 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:41.074455 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:41.074506 (+ 51us) server_negotiation.cc:407] Connection header received
0927 16:39:41.075054 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:41.075069 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:41.075509 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:41.075820 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:41.076575 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:41.076579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:41.076608 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:39:41.076617 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:41.076664 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:39:41.076679 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:41.076811 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:41.076904 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 16:39:41.077111 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":112,"threads_started":1}
I20260927 16:39:41.080611 10204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 976f70b6a7f24f18bc380e0a5555b88c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976f70b6a7f24f18bc380e0a5555b88c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33775 } }
I20260927 16:39:41.080817 10204 raft_consensus.cc:374] T 00000000000000000000000000000000 P 976f70b6a7f24f18bc380e0a5555b88c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:41.080892 10204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 976f70b6a7f24f18bc380e0a5555b88c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 976f70b6a7f24f18bc380e0a5555b88c, State: Initialized, Role: FOLLOWER
I20260927 16:39:41.081262 10204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 976f70b6a7f24f18bc380e0a5555b88c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976f70b6a7f24f18bc380e0a5555b88c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33775 } }
I20260927 16:39:41.081406 10204 raft_consensus.cc:388] T 00000000000000000000000000000000 P 976f70b6a7f24f18bc380e0a5555b88c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:39:41.081485 10204 raft_consensus.cc:482] T 00000000000000000000000000000000 P 976f70b6a7f24f18bc380e0a5555b88c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:39:41.081591 10204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 976f70b6a7f24f18bc380e0a5555b88c [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:41.082336 10204 raft_consensus.cc:504] T 00000000000000000000000000000000 P 976f70b6a7f24f18bc380e0a5555b88c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976f70b6a7f24f18bc380e0a5555b88c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33775 } }
I20260927 16:39:41.082669 10204 leader_election.cc:302] T 00000000000000000000000000000000 P 976f70b6a7f24f18bc380e0a5555b88c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 976f70b6a7f24f18bc380e0a5555b88c; no voters:
I20260927 16:39:41.082943 10204 leader_election.cc:288] T 00000000000000000000000000000000 P 976f70b6a7f24f18bc380e0a5555b88c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:39:41.083051 10209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 976f70b6a7f24f18bc380e0a5555b88c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:39:41.083333 10209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 976f70b6a7f24f18bc380e0a5555b88c [term 1 LEADER]: Becoming Leader. State: Replica: 976f70b6a7f24f18bc380e0a5555b88c, State: Running, Role: LEADER
I20260927 16:39:41.083657 10209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 976f70b6a7f24f18bc380e0a5555b88c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976f70b6a7f24f18bc380e0a5555b88c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33775 } }
I20260927 16:39:41.084057 10204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 976f70b6a7f24f18bc380e0a5555b88c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:39:41.085338 10211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 976f70b6a7f24f18bc380e0a5555b88c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 976f70b6a7f24f18bc380e0a5555b88c. Latest consensus state: current_term: 1 leader_uuid: "976f70b6a7f24f18bc380e0a5555b88c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976f70b6a7f24f18bc380e0a5555b88c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33775 } } }
I20260927 16:39:41.085336 10210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 976f70b6a7f24f18bc380e0a5555b88c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "976f70b6a7f24f18bc380e0a5555b88c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "976f70b6a7f24f18bc380e0a5555b88c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33775 } } }
I20260927 16:39:41.085480 10211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 976f70b6a7f24f18bc380e0a5555b88c [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:41.085480 10210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 976f70b6a7f24f18bc380e0a5555b88c [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:41.085943 10220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:39:41.088775 10220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:39:41.093792 10220 catalog_manager.cc:1347] Generated new cluster ID: 1fc8a0b6b2a74b219cc5f9851f17f50d
I20260927 16:39:41.093864 10220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:39:41.105516 10220 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:39:41.105783 10220 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:39:41.111531 10220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 976f70b6a7f24f18bc380e0a5555b88c: Generated new TSK 0
I20260927 16:39:41.112087 10220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:39:41.122318 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33775
--builtin_ntp_servers=127.0.98.20:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 16:39:41.252138 10228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:41.252364 10228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:41.252400 10228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:41.256387 10228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:41.256453 10228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:41.256522 10228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:39:41.261276 10228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:41.262183 10228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:41.263473 10228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:41.272372 10233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:41.272385 10236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:41.272554 10228 server_base.cc:1034] running on GCE node
W20260927 16:39:41.272405 10234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:41.273075 10228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:41.273666 10228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:41.274818 10228 hybrid_clock.cc:648] HybridClock initialized: now 1790527181274791 us; error 42 us; skew 500 ppm
I20260927 16:39:41.276631 10228 webserver.cc:460] Webserver started at http://127.0.98.1:40971/ using document root <none> and password file <none>
I20260927 16:39:41.277212 10228 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:41.277274 10228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:41.277529 10228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:41.279507 10228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/data/instance:
uuid: "64b54e28fbd342088644aa9491e39915"
format_stamp: "Formatted at 2026-09-27 16:39:41 on dist-test-slave-z40b"
server_key: "364bd85c88bd7a30056b506f3d5a2280"
server_key_iv: "b252dc47f8419c4d05dcb8a33e2ccd0f"
server_key_version: "encryptionkey@0"
I20260927 16:39:41.280058 10228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/wal/instance:
uuid: "64b54e28fbd342088644aa9491e39915"
format_stamp: "Formatted at 2026-09-27 16:39:41 on dist-test-slave-z40b"
server_key: "364bd85c88bd7a30056b506f3d5a2280"
server_key_iv: "b252dc47f8419c4d05dcb8a33e2ccd0f"
server_key_version: "encryptionkey@0"
I20260927 16:39:41.283569 10228 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 16:39:41.285970 10242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:41.287330 10228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:39:41.287439 10228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "64b54e28fbd342088644aa9491e39915"
format_stamp: "Formatted at 2026-09-27 16:39:41 on dist-test-slave-z40b"
server_key: "364bd85c88bd7a30056b506f3d5a2280"
server_key_iv: "b252dc47f8419c4d05dcb8a33e2ccd0f"
server_key_version: "encryptionkey@0"
I20260927 16:39:41.287536 10228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:41.291880 10228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:41.292438 10228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:41.292615 10228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:41.293128 10228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:41.293977 10228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:41.294021 10228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:41.294082 10228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:41.294112 10228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:41.302498 10228 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34495
I20260927 16:39:41.302563 10355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34495 every 8 connection(s)
I20260927 16:39:41.303454 10228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:39:41.308478 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 10228
I20260927 16:39:41.308621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:39:41.308972 392 external_mini_cluster.cc:1428] Setting key 1c61f276a297501a2f417a45177008aa
I20260927 16:39:41.309136 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:41.305105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43077 (local address 127.0.98.62:33775)
0927 16:39:41.305234 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:41.305237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:41.306135 (+ 898us) server_negotiation.cc:407] Connection header received
0927 16:39:41.307450 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:41.307452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:41.307490 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:41.307561 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:41.308622 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:41.308628 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:41.308634 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:39:41.308650 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:41.308674 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:39:41.308684 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:41.308801 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:41.308945 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 16:39:41.308988 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:39:41.310077 10358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:41.305415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33775 (local address 127.0.98.1:43077)
0927 16:39:41.306016 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:41.306052 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:41.307189 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:41.307705 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:41.307711 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:41.307797 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:41.308353 (+ 556us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:39:41.308354 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:41.308390 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:39:41.308450 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:41.308790 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:41.308797 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:41.308803 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 16:39:41.308964 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 16:39:41.309245 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":113,"threads_started":1}
I20260927 16:39:41.311079 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33775
--builtin_ntp_servers=127.0.98.20:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 16:39:41.311165 10356 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33775
I20260927 16:39:41.311391 10356 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:41.311772 10356 heartbeater.cc:511] Master 127.0.98.62:33775 requested a full tablet report, sending...
I20260927 16:39:41.313396 10168 ts_manager.cc:194] Registered new tserver with Master: 64b54e28fbd342088644aa9491e39915 (127.0.98.1:34495)
W20260927 16:39:41.415037 10359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:41.415290 10359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:41.415328 10359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:41.419123 10359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:41.419162 10359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:41.419245 10359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:39:41.423586 10359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:41.424415 10359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:41.425625 10359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:41.434343 10367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:41.434409 10364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:41.434384 10365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:41.434760 10359 server_base.cc:1034] running on GCE node
I20260927 16:39:41.435106 10359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:41.435711 10359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:41.436864 10359 hybrid_clock.cc:648] HybridClock initialized: now 1790527181436836 us; error 38 us; skew 500 ppm
I20260927 16:39:41.438493 10359 webserver.cc:460] Webserver started at http://127.0.98.2:45143/ using document root <none> and password file <none>
I20260927 16:39:41.438977 10359 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:41.439033 10359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:41.439313 10359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:41.441042 10359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/data/instance:
uuid: "357ee914993a426db274797b4029afb2"
format_stamp: "Formatted at 2026-09-27 16:39:41 on dist-test-slave-z40b"
server_key: "4d5cb730ba15beba2c737cc9af21bd00"
server_key_iv: "399737066be48791b9f6b9e9f33de335"
server_key_version: "encryptionkey@0"
I20260927 16:39:41.441555 10359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/wal/instance:
uuid: "357ee914993a426db274797b4029afb2"
format_stamp: "Formatted at 2026-09-27 16:39:41 on dist-test-slave-z40b"
server_key: "4d5cb730ba15beba2c737cc9af21bd00"
server_key_iv: "399737066be48791b9f6b9e9f33de335"
server_key_version: "encryptionkey@0"
I20260927 16:39:41.444895 10359 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:39:41.447009 10373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:41.448168 10359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:39:41.448290 10359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "357ee914993a426db274797b4029afb2"
format_stamp: "Formatted at 2026-09-27 16:39:41 on dist-test-slave-z40b"
server_key: "4d5cb730ba15beba2c737cc9af21bd00"
server_key_iv: "399737066be48791b9f6b9e9f33de335"
server_key_version: "encryptionkey@0"
I20260927 16:39:41.448385 10359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:41.452752 10359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:41.453274 10359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:41.453451 10359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:41.453943 10359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:41.454730 10359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:41.454772 10359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:41.454854 10359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:41.454890 10359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:41.462821 10359 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39305
I20260927 16:39:41.462841 10486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39305 every 8 connection(s)
I20260927 16:39:41.463622 10359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:39:41.467047 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 10359
I20260927 16:39:41.467175 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:39:41.467476 392 external_mini_cluster.cc:1428] Setting key 67769d1a903f9490065956e3850b972a
I20260927 16:39:41.469331 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:41.465142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38519 (local address 127.0.98.62:33775)
0927 16:39:41.465266 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:41.465270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:41.466111 (+ 841us) server_negotiation.cc:407] Connection header received
0927 16:39:41.467517 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:41.467521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:41.467566 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:41.467672 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:41.468866 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:41.468870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:41.468872 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:39:41.468886 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:41.468908 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:39:41.468917 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:41.469027 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:41.469167 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 16:39:41.469207 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 16:39:41.469455 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33775
--builtin_ntp_servers=127.0.98.20:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 16:39:41.470122 10489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:41.465419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33775 (local address 127.0.98.2:38519)
0927 16:39:41.465980 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:41.466013 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:41.467327 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:41.467850 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:41.467860 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:41.467964 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:41.468543 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:39:41.468544 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:41.468577 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:39:41.468635 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:41.469015 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:41.469023 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:41.469028 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 16:39:41.469202 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 16:39:41.469420 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":96,"threads_started":1}
I20260927 16:39:41.471289 10487 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33775
I20260927 16:39:41.471499 10487 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:41.471833 10487 heartbeater.cc:511] Master 127.0.98.62:33775 requested a full tablet report, sending...
I20260927 16:39:41.472788 10168 ts_manager.cc:194] Registered new tserver with Master: 357ee914993a426db274797b4029afb2 (127.0.98.2:39305)
W20260927 16:39:41.575713 10490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:41.575928 10490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:41.575963 10490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:41.580128 10490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:41.580170 10490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:41.580253 10490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:39:41.584705 10490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:41.585695 10490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:41.586969 10490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:41.595676 10496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:41.595793 10490 server_base.cc:1034] running on GCE node
W20260927 16:39:41.595695 10495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:41.595652 10498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:41.596194 10490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:41.596750 10490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:41.597920 10490 hybrid_clock.cc:648] HybridClock initialized: now 1790527181597894 us; error 38 us; skew 500 ppm
I20260927 16:39:41.599637 10490 webserver.cc:460] Webserver started at http://127.0.98.3:42803/ using document root <none> and password file <none>
I20260927 16:39:41.600153 10490 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:41.600211 10490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:41.600448 10490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:41.602304 10490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/data/instance:
uuid: "b34859efae3b47eca73c3bdd445802d5"
format_stamp: "Formatted at 2026-09-27 16:39:41 on dist-test-slave-z40b"
server_key: "f566301dacdd77d4a41cd29a8bec5a48"
server_key_iv: "f0389e422f2e50e9d22748d06b768215"
server_key_version: "encryptionkey@0"
I20260927 16:39:41.602833 10490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b34859efae3b47eca73c3bdd445802d5"
format_stamp: "Formatted at 2026-09-27 16:39:41 on dist-test-slave-z40b"
server_key: "f566301dacdd77d4a41cd29a8bec5a48"
server_key_iv: "f0389e422f2e50e9d22748d06b768215"
server_key_version: "encryptionkey@0"
I20260927 16:39:41.606138 10490 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:39:41.608315 10504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:41.609464 10490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:39:41.609586 10490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "b34859efae3b47eca73c3bdd445802d5"
format_stamp: "Formatted at 2026-09-27 16:39:41 on dist-test-slave-z40b"
server_key: "f566301dacdd77d4a41cd29a8bec5a48"
server_key_iv: "f0389e422f2e50e9d22748d06b768215"
server_key_version: "encryptionkey@0"
I20260927 16:39:41.609681 10490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:41.613909 10490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:41.614449 10490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:41.614619 10490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:41.615135 10490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:41.615947 10490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:41.615991 10490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:41.616053 10490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:41.616091 10490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:41.624048 10490 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37435
I20260927 16:39:41.624081 10617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37435 every 8 connection(s)
I20260927 16:39:41.624758 10490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:39:41.625617 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 10490
I20260927 16:39:41.625697 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790527121515808-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:39:41.625963 392 external_mini_cluster.cc:1428] Setting key df4c1a3786f75dfe8e36f8b0a1c67062
I20260927 16:39:41.630226 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:41.626397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58701 (local address 127.0.98.62:33775)
0927 16:39:41.626530 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:41.626534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:41.627697 (+ 1163us) server_negotiation.cc:407] Connection header received
0927 16:39:41.628908 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:41.628910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:41.628945 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:41.629006 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:41.629899 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:41.629902 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:41.629904 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:39:41.629912 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:41.629930 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:39:41.629937 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:41.629999 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:41.630084 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 16:39:41.630117 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 16:39:41.631042 10620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:41.626863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33775 (local address 127.0.98.3:58701)
0927 16:39:41.627554 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:41.627599 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:41.628721 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:41.629126 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:41.629131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:41.629205 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:41.629649 (+ 444us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:39:41.629651 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:41.629684 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:39:41.629772 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:41.629988 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:41.629991 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:41.629994 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 16:39:41.630096 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 16:39:41.630274 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":121,"threads_started":1}
I20260927 16:39:41.632040 10618 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33775
I20260927 16:39:41.632270 10618 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:41.632629 10618 heartbeater.cc:511] Master 127.0.98.62:33775 requested a full tablet report, sending...
I20260927 16:39:41.633517 10168 ts_manager.cc:194] Registered new tserver with Master: b34859efae3b47eca73c3bdd445802d5 (127.0.98.3:37435)
I20260927 16:39:41.639577 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 16:39:41.642145 10207 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:39:41.641313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53584 (local address 127.0.98.62:33775)
0927 16:39:41.641431 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:41.641436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:41.641607 (+ 171us) server_negotiation.cc:407] Connection header received
0927 16:39:41.641737 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:41.641740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:41.641777 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:41.641856 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:41.642055 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53584: BlockingRecv error: recv got EOF from 127.0.0.1:53584 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 16:39:41.642473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 10228
I20260927 16:39:41.647692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 10359
I20260927 16:39:41.652835 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 10490
I20260927 16:39:41.657760 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 10137
2026-09-27T16:39:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (767 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:39:41 dist-test-slave-z40b krb5kdc[10628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:39:41 dist-test-slave-z40b krb5kdc[10628](info): set up 2 sockets
Sep 27 16:39:41 dist-test-slave-z40b krb5kdc[10628](info): commencing operation
krb5kdc: starting...
W20260927 16:39:42.668220 392 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:39:42 dist-test-slave-z40b krb5kdc[10628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:39:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:39:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:39:42.780125 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45347
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:42.885051 10641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:42.885300 10641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:42.885360 10641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:42.889240 10641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:39:42.889317 10641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:42.889355 10641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:42.889415 10641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:39:42.889449 10641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:39:42.894851 10641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:42.895828 10641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:42.896800 10641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:42.904726 10646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:42.904800 10649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:42.904798 10647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:42.905004 10641 server_base.cc:1034] running on GCE node
I20260927 16:39:42.905539 10641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:42.906616 10641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:42.907795 10641 hybrid_clock.cc:648] HybridClock initialized: now 1790527182907771 us; error 44 us; skew 500 ppm
Sep 27 16:39:42 dist-test-slave-z40b krb5kdc[10628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:42.914011 10641 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:39:42.914870 10641 webserver.cc:460] Webserver started at http://127.0.98.62:45477/ using document root <none> and password file <none>
I20260927 16:39:42.915488 10641 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:42.915577 10641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:42.915794 10641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:42.917713 10641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "8d1ac4e9c11548df87cec6ce59d441d9"
format_stamp: "Formatted at 2026-09-27 16:39:42 on dist-test-slave-z40b"
server_key: "8be0ed5729857b76a4d265eb3bb42996"
server_key_iv: "f8cebe9f6c448ebc23410588ebaa3914"
server_key_version: "encryptionkey@0"
I20260927 16:39:42.918241 10641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "8d1ac4e9c11548df87cec6ce59d441d9"
format_stamp: "Formatted at 2026-09-27 16:39:42 on dist-test-slave-z40b"
server_key: "8be0ed5729857b76a4d265eb3bb42996"
server_key_iv: "f8cebe9f6c448ebc23410588ebaa3914"
server_key_version: "encryptionkey@0"
I20260927 16:39:42.921725 10641 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:39:42.923952 10656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:42.925241 10641 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:39:42.925377 10641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "8d1ac4e9c11548df87cec6ce59d441d9"
format_stamp: "Formatted at 2026-09-27 16:39:42 on dist-test-slave-z40b"
server_key: "8be0ed5729857b76a4d265eb3bb42996"
server_key_iv: "f8cebe9f6c448ebc23410588ebaa3914"
server_key_version: "encryptionkey@0"
I20260927 16:39:42.925482 10641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:42.937146 10641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:42.940301 10641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:42.940495 10641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:42.946830 10641 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45347
I20260927 16:39:42.947095 10708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45347 every 8 connection(s)
I20260927 16:39:42.947762 10641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:39:42.950311 10709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:42.954962 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 10641
I20260927 16:39:42.955096 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:39:42.955404 392 external_mini_cluster.cc:1428] Setting key a1cac77d03af515c8ef84fc1119e03bc
I20260927 16:39:42.955595 10709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d1ac4e9c11548df87cec6ce59d441d9: Bootstrap starting.
I20260927 16:39:42.957935 10709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d1ac4e9c11548df87cec6ce59d441d9: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:42.958758 10709 log.cc:824] T 00000000000000000000000000000000 P 8d1ac4e9c11548df87cec6ce59d441d9: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:42.960745 10709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d1ac4e9c11548df87cec6ce59d441d9: No bootstrap required, opened a new log
Sep 27 16:39:42 dist-test-slave-z40b krb5kdc[10628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:39:42.965018 10709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d1ac4e9c11548df87cec6ce59d441d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1ac4e9c11548df87cec6ce59d441d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45347 } }
I20260927 16:39:42.965385 10709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d1ac4e9c11548df87cec6ce59d441d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:42.965447 10709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d1ac4e9c11548df87cec6ce59d441d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d1ac4e9c11548df87cec6ce59d441d9, State: Initialized, Role: FOLLOWER
I20260927 16:39:42.965905 10709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d1ac4e9c11548df87cec6ce59d441d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1ac4e9c11548df87cec6ce59d441d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45347 } }
I20260927 16:39:42.966074 10709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d1ac4e9c11548df87cec6ce59d441d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:39:42.966156 10709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d1ac4e9c11548df87cec6ce59d441d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:39:42.966274 10709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d1ac4e9c11548df87cec6ce59d441d9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:42.967087 10709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d1ac4e9c11548df87cec6ce59d441d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1ac4e9c11548df87cec6ce59d441d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45347 } }
I20260927 16:39:42.967428 10709 leader_election.cc:302] T 00000000000000000000000000000000 P 8d1ac4e9c11548df87cec6ce59d441d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d1ac4e9c11548df87cec6ce59d441d9; no voters:
I20260927 16:39:42.967747 10709 leader_election.cc:288] T 00000000000000000000000000000000 P 8d1ac4e9c11548df87cec6ce59d441d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:39:42.967808 10714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d1ac4e9c11548df87cec6ce59d441d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:39:42.967976 10714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d1ac4e9c11548df87cec6ce59d441d9 [term 1 LEADER]: Becoming Leader. State: Replica: 8d1ac4e9c11548df87cec6ce59d441d9, State: Running, Role: LEADER
I20260927 16:39:42.968333 10714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d1ac4e9c11548df87cec6ce59d441d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1ac4e9c11548df87cec6ce59d441d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45347 } }
I20260927 16:39:42.968905 10709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d1ac4e9c11548df87cec6ce59d441d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:39:42.970002 10715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d1ac4e9c11548df87cec6ce59d441d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d1ac4e9c11548df87cec6ce59d441d9. Latest consensus state: current_term: 1 leader_uuid: "8d1ac4e9c11548df87cec6ce59d441d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1ac4e9c11548df87cec6ce59d441d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45347 } } }
I20260927 16:39:42.970124 10715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d1ac4e9c11548df87cec6ce59d441d9 [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:42.970407 10716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d1ac4e9c11548df87cec6ce59d441d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d1ac4e9c11548df87cec6ce59d441d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1ac4e9c11548df87cec6ce59d441d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45347 } } }
I20260927 16:39:42.970516 10723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:39:42.970552 10712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:42.956718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56638 (local address 127.0.98.62:45347)
0927 16:39:42.957084 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:42.957105 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:42.957150 (+ 45us) server_negotiation.cc:407] Connection header received
0927 16:39:42.957738 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:42.957753 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:42.958094 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:42.958348 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:42.959333 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:42.960386 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:42.961511 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:42.961942 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:42.962054 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:42.964823 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:42.964827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:42.964839 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:42.964864 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:42.966939 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:42.967445 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:42.967451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:42.967457 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:42.967587 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:42.967928 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:42.967931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:42.967933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:42.968394 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:42.969096 (+ 702us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:42.969204 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 16:39:42.969518 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":121,"threads_started":1}
I20260927 16:39:42.970546 10716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d1ac4e9c11548df87cec6ce59d441d9 [sys.catalog]: This master's current role is: LEADER
I20260927 16:39:42.973554 10723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:39:42.980645 10723 catalog_manager.cc:1347] Generated new cluster ID: ef19630721b7499bb89d0eeab8380e5f
I20260927 16:39:42.980711 10723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:39:42.987551 10723 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:39:42.988336 10723 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:39:42.997292 10723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d1ac4e9c11548df87cec6ce59d441d9: Generated new TSK 0
I20260927 16:39:42.998008 10723 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:39:43.028525 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45347
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:43.130780 10735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:43.130997 10735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:43.131036 10735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:43.134822 10735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:43.134871 10735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:43.134925 10735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:39:43.139577 10735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:43.140417 10735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:43.141271 10735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:43.150142 10741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:43.150143 10740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:43.150265 10735 server_base.cc:1034] running on GCE node
W20260927 16:39:43.150143 10743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:43.150750 10735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:43.151383 10735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:43.152529 10735 hybrid_clock.cc:648] HybridClock initialized: now 1790527183152501 us; error 40 us; skew 500 ppm
Sep 27 16:39:43 dist-test-slave-z40b krb5kdc[10628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:43.155089 10735 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:39:43.155925 10735 webserver.cc:460] Webserver started at http://127.0.98.1:33365/ using document root <none> and password file <none>
I20260927 16:39:43.156486 10735 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:43.156574 10735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:43.156796 10735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:43.158669 10735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-0/data/instance:
uuid: "f5e6c5538fa84a4b90380eb5c2784a3b"
format_stamp: "Formatted at 2026-09-27 16:39:43 on dist-test-slave-z40b"
server_key: "0e8ac610675ae6ce7a65aba08c375b11"
server_key_iv: "5554a31e75e664c692a7b3a9e6b26af0"
server_key_version: "encryptionkey@0"
I20260927 16:39:43.159243 10735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f5e6c5538fa84a4b90380eb5c2784a3b"
format_stamp: "Formatted at 2026-09-27 16:39:43 on dist-test-slave-z40b"
server_key: "0e8ac610675ae6ce7a65aba08c375b11"
server_key_iv: "5554a31e75e664c692a7b3a9e6b26af0"
server_key_version: "encryptionkey@0"
I20260927 16:39:43.162515 10735 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:39:43.164777 10750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:43.165845 10735 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:39:43.165961 10735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "f5e6c5538fa84a4b90380eb5c2784a3b"
format_stamp: "Formatted at 2026-09-27 16:39:43 on dist-test-slave-z40b"
server_key: "0e8ac610675ae6ce7a65aba08c375b11"
server_key_iv: "5554a31e75e664c692a7b3a9e6b26af0"
server_key_version: "encryptionkey@0"
I20260927 16:39:43.166054 10735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:43.183122 10735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:43.185688 10735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:43.185878 10735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:43.186447 10735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:43.187230 10735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:43.187309 10735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:43.187376 10735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:43.187410 10735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:43.196352 10735 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44207
I20260927 16:39:43.196394 10863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44207 every 8 connection(s)
I20260927 16:39:43.197142 10735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:39:43.203744 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 10735
I20260927 16:39:43.203866 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:39:43.204200 392 external_mini_cluster.cc:1428] Setting key 24a0ec3a4d70cce4504f818aa61d713b
Sep 27 16:39:43 dist-test-slave-z40b krb5kdc[10628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:39:43.210659 10712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:43.198758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58367 (local address 127.0.98.62:45347)
0927 16:39:43.198897 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:43.198901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:43.199947 (+ 1046us) server_negotiation.cc:407] Connection header received
0927 16:39:43.200718 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:43.200721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:43.200761 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:43.200839 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:43.202027 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.202542 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.203652 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.203986 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.204064 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:43.206834 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:43.206836 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:43.206838 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:43.206858 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:43.208599 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:43.209178 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:43.209180 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:43.209182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:43.209249 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:43.209638 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:43.209640 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:43.209641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:43.209790 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:43.209862 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:43.210458 (+ 596us) server_negotiation.cc:299] Negotiation successful
0927 16:39:43.210547 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 16:39:43.211892 10866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:43.199159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45347 (local address 127.0.98.1:58367)
0927 16:39:43.199814 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:43.199854 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:43.200554 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:43.201047 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:43.201057 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:43.201473 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:43.201905 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:43.201918 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.202644 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.202646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:43.203539 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:43.203552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.204131 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.204136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:43.204317 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:43.204910 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:43.204937 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:43.206675 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:43.208710 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:43.208718 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:43.208733 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:43.209087 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:43.209361 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:43.209367 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:43.209372 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:43.209548 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:43.209899 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:43.209907 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:43.210209 (+ 302us) client_negotiation.cc:769] Sending connection context
0927 16:39:43.210421 (+ 212us) client_negotiation.cc:240] Negotiation successful
0927 16:39:43.210755 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":107,"threads_started":1}
I20260927 16:39:43.213068 10864 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45347
I20260927 16:39:43.213299 10864 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:43.213871 10864 heartbeater.cc:511] Master 127.0.98.62:45347 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:39:43.215340 10673 ts_manager.cc:194] Registered new tserver with Master: f5e6c5538fa84a4b90380eb5c2784a3b (127.0.98.1:44207)
I20260927 16:39:43.216550 10673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:39:43.230283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45347
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:43.335307 10869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:43.335554 10869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:43.335593 10869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:43.339156 10869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:43.339246 10869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:43.339300 10869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:39:43.344308 10869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:43.345259 10869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:43.346175 10869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:43.355358 10877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:43.355438 10869 server_base.cc:1034] running on GCE node
W20260927 16:39:43.355358 10874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:43.355402 10875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:43.355906 10869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:43.356556 10869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:43.357756 10869 hybrid_clock.cc:648] HybridClock initialized: now 1790527183357744 us; error 24 us; skew 500 ppm
Sep 27 16:39:43 dist-test-slave-z40b krb5kdc[10628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:43.360464 10869 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:39:43.361317 10869 webserver.cc:460] Webserver started at http://127.0.98.2:40571/ using document root <none> and password file <none>
I20260927 16:39:43.361850 10869 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:43.361939 10869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:43.362161 10869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:43.364181 10869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-1/data/instance:
uuid: "8db9826a2651482095e9921f3d703990"
format_stamp: "Formatted at 2026-09-27 16:39:43 on dist-test-slave-z40b"
server_key: "0b7c37214e41a78a03a9b683de8426d2"
server_key_iv: "4f5b65b80a20cd4633dab81dafc15bc9"
server_key_version: "encryptionkey@0"
I20260927 16:39:43.364710 10869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8db9826a2651482095e9921f3d703990"
format_stamp: "Formatted at 2026-09-27 16:39:43 on dist-test-slave-z40b"
server_key: "0b7c37214e41a78a03a9b683de8426d2"
server_key_iv: "4f5b65b80a20cd4633dab81dafc15bc9"
server_key_version: "encryptionkey@0"
I20260927 16:39:43.368135 10869 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 16:39:43.370630 10884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:43.371814 10869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:39:43.371946 10869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "8db9826a2651482095e9921f3d703990"
format_stamp: "Formatted at 2026-09-27 16:39:43 on dist-test-slave-z40b"
server_key: "0b7c37214e41a78a03a9b683de8426d2"
server_key_iv: "4f5b65b80a20cd4633dab81dafc15bc9"
server_key_version: "encryptionkey@0"
I20260927 16:39:43.372046 10869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:43.384481 10869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:43.387081 10869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:43.387306 10869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:43.387858 10869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:43.388603 10869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:43.388674 10869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:43.388732 10869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:43.388782 10869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:43.397199 10869 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42847
I20260927 16:39:43.397221 10997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42847 every 8 connection(s)
I20260927 16:39:43.398018 10869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:39:43.405608 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 10869
I20260927 16:39:43.405699 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:39:43.406034 392 external_mini_cluster.cc:1428] Setting key 21561d0b646b8da029839ca9f4ae0cf8
Sep 27 16:39:43 dist-test-slave-z40b krb5kdc[10628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:39:43.411562 10712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:43.399695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55027 (local address 127.0.98.62:45347)
0927 16:39:43.399821 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:43.399825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:43.400921 (+ 1096us) server_negotiation.cc:407] Connection header received
0927 16:39:43.401901 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:43.401904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:43.401961 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:43.402062 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:43.403090 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.403678 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.404752 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.404979 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.405084 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:43.407680 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:43.407685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:43.407688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:43.407719 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:43.409339 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:43.409921 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:43.409924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:43.409925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:43.409993 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:43.410364 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:43.410366 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:43.410367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:43.410612 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:43.410712 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:43.411168 (+ 456us) server_negotiation.cc:299] Negotiation successful
0927 16:39:43.411325 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 16:39:43.412073 11000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:43.399974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45347 (local address 127.0.98.2:55027)
0927 16:39:43.400784 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:43.400823 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:43.401704 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:43.402215 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:43.402221 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:43.402535 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:43.402948 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:43.402959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.403798 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.403803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:43.404630 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:43.404638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.405123 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.405127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:43.405287 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:43.405920 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:43.405942 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:43.407534 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:43.409446 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:43.409454 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:43.409468 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:43.409816 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:43.410078 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:43.410084 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:43.410088 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:43.410267 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:43.410710 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:43.410715 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:43.410911 (+ 196us) client_negotiation.cc:769] Sending connection context
0927 16:39:43.411073 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 16:39:43.411367 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":515,"threads_started":1}
I20260927 16:39:43.413130 10998 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45347
I20260927 16:39:43.413371 10998 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:43.413944 10998 heartbeater.cc:511] Master 127.0.98.62:45347 requested a full tablet report, sending...
I20260927 16:39:43.415019 10673 ts_manager.cc:194] Registered new tserver with Master: 8db9826a2651482095e9921f3d703990 (127.0.98.2:42847)
I20260927 16:39:43.415649 10673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55027
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:39:43.432111 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45347
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:39:43.535851 11003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:39:43.536100 11003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:39:43.536172 11003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:39:43.539935 11003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:39:43.540006 11003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:39:43.540091 11003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:39:43.544905 11003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:39:43.545792 11003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:39:43.546880 11003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:39:43.556274 11008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:43.556383 11003 server_base.cc:1034] running on GCE node
W20260927 16:39:43.556223 11009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:39:43.556320 11011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:39:43.556854 11003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:39:43.557404 11003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:39:43.558583 11003 hybrid_clock.cc:648] HybridClock initialized: now 1790527183558573 us; error 27 us; skew 500 ppm
Sep 27 16:39:43 dist-test-slave-z40b krb5kdc[10628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:39:43.561120 11003 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:39:43.561880 11003 webserver.cc:460] Webserver started at http://127.0.98.3:45341/ using document root <none> and password file <none>
I20260927 16:39:43.562381 11003 fs_manager.cc:359] Metadata directory not provided
I20260927 16:39:43.562464 11003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:39:43.562671 11003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:39:43.564589 11003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-2/data/instance:
uuid: "f8582af6bbbd4a589a1fabcfa16b34b9"
format_stamp: "Formatted at 2026-09-27 16:39:43 on dist-test-slave-z40b"
server_key: "7b34e76ffda389f6285382c7e9fb65b3"
server_key_iv: "2ec7f27969ba1a0827232c1cbca18cb7"
server_key_version: "encryptionkey@0"
I20260927 16:39:43.565111 11003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f8582af6bbbd4a589a1fabcfa16b34b9"
format_stamp: "Formatted at 2026-09-27 16:39:43 on dist-test-slave-z40b"
server_key: "7b34e76ffda389f6285382c7e9fb65b3"
server_key_iv: "2ec7f27969ba1a0827232c1cbca18cb7"
server_key_version: "encryptionkey@0"
I20260927 16:39:43.568509 11003 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:39:43.570876 11018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:43.572114 11003 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:39:43.572233 11003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "f8582af6bbbd4a589a1fabcfa16b34b9"
format_stamp: "Formatted at 2026-09-27 16:39:43 on dist-test-slave-z40b"
server_key: "7b34e76ffda389f6285382c7e9fb65b3"
server_key_iv: "2ec7f27969ba1a0827232c1cbca18cb7"
server_key_version: "encryptionkey@0"
I20260927 16:39:43.572340 11003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:39:43.590219 11003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:39:43.593315 11003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:39:43.593506 11003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:39:43.594043 11003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:39:43.594779 11003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:39:43.594830 11003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:43.594894 11003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:39:43.594934 11003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:39:43.603125 11003 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36515
I20260927 16:39:43.603197 11131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36515 every 8 connection(s)
I20260927 16:39:43.603971 11003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:39:43.607446 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 11003
I20260927 16:39:43.607525 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:39:43.607770 392 external_mini_cluster.cc:1428] Setting key 511ecd45d789a3dc0279a8edc3d14f99
Sep 27 16:39:43 dist-test-slave-z40b krb5kdc[10628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:39:43.616664 10712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:43.605551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36157 (local address 127.0.98.62:45347)
0927 16:39:43.605688 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:43.605692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:43.606706 (+ 1014us) server_negotiation.cc:407] Connection header received
0927 16:39:43.607618 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:43.607621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:43.607677 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:43.607782 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:43.609004 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.609743 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.610779 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.610990 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.611060 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:43.613242 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:43.613245 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:43.613247 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:43.613268 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:43.614705 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:43.615198 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:43.615203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:43.615205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:43.615320 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:43.615643 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:43.615648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:43.615649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:43.615837 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:43.615938 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:43.616398 (+ 460us) server_negotiation.cc:299] Negotiation successful
0927 16:39:43.616508 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:39:43.617398 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:43.605930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45347 (local address 127.0.98.3:36157)
0927 16:39:43.606562 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:43.606599 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:43.607400 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:43.607943 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:43.607950 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:43.608333 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:39:43.608852 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:43.608865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.609867 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.609870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:43.610687 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:43.610694 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.611106 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.611108 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:43.611296 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:43.611763 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:39:43.611777 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:39:43.613084 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:39:43.614829 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:43.614834 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:43.614847 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:43.615087 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:43.615410 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:39:43.615414 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:39:43.615416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:39:43.615544 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:39:43.615935 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:39:43.615941 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:39:43.616161 (+ 220us) client_negotiation.cc:769] Sending connection context
0927 16:39:43.616356 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 16:39:43.616633 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":113,"threads_started":1}
I20260927 16:39:43.618440 11132 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45347
I20260927 16:39:43.618630 11132 heartbeater.cc:464] Registering TS with master...
I20260927 16:39:43.619017 11132 heartbeater.cc:511] Master 127.0.98.62:45347 requested a full tablet report, sending...
I20260927 16:39:43.619948 10673 ts_manager.cc:194] Registered new tserver with Master: f8582af6bbbd4a589a1fabcfa16b34b9 (127.0.98.3:36515)
I20260927 16:39:43.620482 10673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36157
I20260927 16:39:43.621336 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:39:43.629515 10712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:43.623366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56640 (local address 127.0.98.62:45347)
0927 16:39:43.623472 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:43.623475 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:43.623643 (+ 168us) server_negotiation.cc:407] Connection header received
0927 16:39:43.623743 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:43.623745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:43.623775 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:43.623833 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:39:43.624408 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.624847 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.625600 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.625790 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.625860 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:43.626962 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:39:43.626965 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:39:43.626967 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:39:43.626986 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:39:43.628159 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:43.628596 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:43.628599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:43.628600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:43.628666 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:39:43.628919 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:39:43.628921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:39:43.628922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:39:43.629089 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:39:43.629181 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:43.629339 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 16:39:43.629420 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 16:39:43.631852 10673 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:39:43.634590 10673 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:39:43.648092 11147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:43.643164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42847 (local address 127.0.0.1:40466)
0927 16:39:43.643569 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:43.643582 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:43.643657 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:43.644392 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:43.644396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:43.644412 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:43.644483 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:43.644487 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.645496 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.645502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:43.646667 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:43.646675 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.647540 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.647543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:43.647697 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:43.647732 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 16:39:43.647823 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 16:39:43.647958 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":29,"thread_start_us":174,"threads_started":1}
I20260927 16:39:43.648198 11143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:43.642089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36515 (local address 127.0.0.1:37956)
0927 16:39:43.642859 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:43.642913 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:43.643035 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:43.643989 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:43.643992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:43.644003 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:43.644278 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:43.644290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.645661 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.645665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:43.646734 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:43.646744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.647807 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.647813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:43.647931 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:43.647949 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 16:39:43.648006 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 16:39:43.648057 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":159,"spinlock_wait_cycles":2816,"thread_start_us":82,"threads_started":1}
I20260927 16:39:43.648456 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:43.642171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37956 (local address 127.0.98.3:36515)
0927 16:39:43.643543 (+ 1372us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:43.643556 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:43.643576 (+ 20us) server_negotiation.cc:407] Connection header received
0927 16:39:43.643643 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:43.643648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:43.643845 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:43.644182 (+ 337us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:43.644387 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.645507 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.646855 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.647690 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.647781 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:43.647920 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:43.648054 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 16:39:43.648135 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1280,"thread_start_us":99,"threads_started":1}
I20260927 16:39:43.648638 11148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:43.643647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40466 (local address 127.0.98.2:42847)
0927 16:39:43.643967 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:43.643978 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:43.643997 (+ 19us) server_negotiation.cc:407] Connection header received
0927 16:39:43.644068 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:43.644073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:43.644240 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:43.644599 (+ 359us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:43.644718 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.645374 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.646787 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.647421 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.647538 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:43.647673 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:43.648414 (+ 741us) server_negotiation.cc:299] Negotiation successful
0927 16:39:43.648498 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":98,"threads_started":1}
I20260927 16:39:43.649426 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:43.642237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44207 (local address 127.0.0.1:47538)
0927 16:39:43.643165 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:43.643177 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:43.643299 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:43.643966 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:43.643976 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:43.643998 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:43.644232 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:43.644242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.645661 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.645665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:43.647960 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:43.647969 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.649082 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.649086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:43.649171 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:43.649191 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 16:39:43.649249 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 16:39:43.649290 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"mutex_wait_us":343,"thread_start_us":81,"threads_started":1}
I20260927 16:39:43.650836 11066 tablet_service.cc:1467] Processing CreateTablet for tablet fbdc798725524341967c066f767a4506 (DEFAULT_TABLE table=test-table [id=d3340e94a29d4681858f12115249d8ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:39:43.650842 10932 tablet_service.cc:1467] Processing CreateTablet for tablet fbdc798725524341967c066f767a4506 (DEFAULT_TABLE table=test-table [id=d3340e94a29d4681858f12115249d8ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:39:43.651707 10932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbdc798725524341967c066f767a4506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:43.651742 11066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbdc798725524341967c066f767a4506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:43.654559 11146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:43.642424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47538 (local address 127.0.98.1:44207)
0927 16:39:43.642702 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:43.642713 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:43.643290 (+ 577us) server_negotiation.cc:407] Connection header received
0927 16:39:43.643575 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:43.643582 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:43.643816 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:43.643971 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:43.644362 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.645507 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.648081 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.648965 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.649065 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:43.654272 (+ 5207us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:43.654334 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 16:39:43.654427 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":104,"threads_started":1}
I20260927 16:39:43.657343 11149 tablet_bootstrap.cc:492] T fbdc798725524341967c066f767a4506 P f8582af6bbbd4a589a1fabcfa16b34b9: Bootstrap starting.
I20260927 16:39:43.657068 10798 tablet_service.cc:1467] Processing CreateTablet for tablet fbdc798725524341967c066f767a4506 (DEFAULT_TABLE table=test-table [id=d3340e94a29d4681858f12115249d8ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:39:43.658031 10798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbdc798725524341967c066f767a4506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:39:43.659860 11149 tablet_bootstrap.cc:654] T fbdc798725524341967c066f767a4506 P f8582af6bbbd4a589a1fabcfa16b34b9: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:43.660898 11149 log.cc:824] T fbdc798725524341967c066f767a4506 P f8582af6bbbd4a589a1fabcfa16b34b9: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:43.661353 11150 tablet_bootstrap.cc:492] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990: Bootstrap starting.
I20260927 16:39:43.662869 11149 tablet_bootstrap.cc:492] T fbdc798725524341967c066f767a4506 P f8582af6bbbd4a589a1fabcfa16b34b9: No bootstrap required, opened a new log
I20260927 16:39:43.663110 11149 ts_tablet_manager.cc:1397] T fbdc798725524341967c066f767a4506 P f8582af6bbbd4a589a1fabcfa16b34b9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:39:43.663854 11150 tablet_bootstrap.cc:654] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:43.664068 11152 tablet_bootstrap.cc:492] T fbdc798725524341967c066f767a4506 P f5e6c5538fa84a4b90380eb5c2784a3b: Bootstrap starting.
I20260927 16:39:43.664734 11150 log.cc:824] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:43.665863 11152 tablet_bootstrap.cc:654] T fbdc798725524341967c066f767a4506 P f5e6c5538fa84a4b90380eb5c2784a3b: Neither blocks nor log segments found. Creating new log.
I20260927 16:39:43.666630 11150 tablet_bootstrap.cc:492] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990: No bootstrap required, opened a new log
I20260927 16:39:43.666815 11150 ts_tablet_manager.cc:1397] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:39:43.666831 11152 log.cc:824] T fbdc798725524341967c066f767a4506 P f5e6c5538fa84a4b90380eb5c2784a3b: Log is configured to *not* fsync() on all Append() calls
I20260927 16:39:43.667016 11149 raft_consensus.cc:348] T fbdc798725524341967c066f767a4506 P f8582af6bbbd4a589a1fabcfa16b34b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8582af6bbbd4a589a1fabcfa16b34b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36515 } } peers { permanent_uuid: "f5e6c5538fa84a4b90380eb5c2784a3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44207 } } peers { permanent_uuid: "8db9826a2651482095e9921f3d703990" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42847 } }
I20260927 16:39:43.667299 11149 raft_consensus.cc:374] T fbdc798725524341967c066f767a4506 P f8582af6bbbd4a589a1fabcfa16b34b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:43.667392 11149 raft_consensus.cc:729] T fbdc798725524341967c066f767a4506 P f8582af6bbbd4a589a1fabcfa16b34b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8582af6bbbd4a589a1fabcfa16b34b9, State: Initialized, Role: FOLLOWER
I20260927 16:39:43.667810 11149 consensus_queue.cc:260] T fbdc798725524341967c066f767a4506 P f8582af6bbbd4a589a1fabcfa16b34b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8582af6bbbd4a589a1fabcfa16b34b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36515 } } peers { permanent_uuid: "f5e6c5538fa84a4b90380eb5c2784a3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44207 } } peers { permanent_uuid: "8db9826a2651482095e9921f3d703990" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42847 } }
I20260927 16:39:43.668586 11152 tablet_bootstrap.cc:492] T fbdc798725524341967c066f767a4506 P f5e6c5538fa84a4b90380eb5c2784a3b: No bootstrap required, opened a new log
I20260927 16:39:43.668728 11149 ts_tablet_manager.cc:1428] T fbdc798725524341967c066f767a4506 P f8582af6bbbd4a589a1fabcfa16b34b9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:39:43.668777 11152 ts_tablet_manager.cc:1397] T fbdc798725524341967c066f767a4506 P f5e6c5538fa84a4b90380eb5c2784a3b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:39:43.669067 11132 heartbeater.cc:503] Master 127.0.98.62:45347 was elected leader, sending a full tablet report...
I20260927 16:39:43.671265 11150 raft_consensus.cc:348] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8582af6bbbd4a589a1fabcfa16b34b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36515 } } peers { permanent_uuid: "f5e6c5538fa84a4b90380eb5c2784a3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44207 } } peers { permanent_uuid: "8db9826a2651482095e9921f3d703990" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42847 } }
I20260927 16:39:43.671495 11150 raft_consensus.cc:374] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:43.671593 11150 raft_consensus.cc:729] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8db9826a2651482095e9921f3d703990, State: Initialized, Role: FOLLOWER
I20260927 16:39:43.672010 11150 consensus_queue.cc:260] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8582af6bbbd4a589a1fabcfa16b34b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36515 } } peers { permanent_uuid: "f5e6c5538fa84a4b90380eb5c2784a3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44207 } } peers { permanent_uuid: "8db9826a2651482095e9921f3d703990" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42847 } }
I20260927 16:39:43.672223 11152 raft_consensus.cc:348] T fbdc798725524341967c066f767a4506 P f5e6c5538fa84a4b90380eb5c2784a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8582af6bbbd4a589a1fabcfa16b34b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36515 } } peers { permanent_uuid: "f5e6c5538fa84a4b90380eb5c2784a3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44207 } } peers { permanent_uuid: "8db9826a2651482095e9921f3d703990" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42847 } }
I20260927 16:39:43.672513 11152 raft_consensus.cc:374] T fbdc798725524341967c066f767a4506 P f5e6c5538fa84a4b90380eb5c2784a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:39:43.672605 10998 heartbeater.cc:503] Master 127.0.98.62:45347 was elected leader, sending a full tablet report...
I20260927 16:39:43.672606 11152 raft_consensus.cc:729] T fbdc798725524341967c066f767a4506 P f5e6c5538fa84a4b90380eb5c2784a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e6c5538fa84a4b90380eb5c2784a3b, State: Initialized, Role: FOLLOWER
I20260927 16:39:43.672925 11150 ts_tablet_manager.cc:1428] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 16:39:43.672997 11152 consensus_queue.cc:260] T fbdc798725524341967c066f767a4506 P f5e6c5538fa84a4b90380eb5c2784a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8582af6bbbd4a589a1fabcfa16b34b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36515 } } peers { permanent_uuid: "f5e6c5538fa84a4b90380eb5c2784a3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44207 } } peers { permanent_uuid: "8db9826a2651482095e9921f3d703990" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42847 } }
I20260927 16:39:43.673659 10864 heartbeater.cc:503] Master 127.0.98.62:45347 was elected leader, sending a full tablet report...
I20260927 16:39:43.673861 11152 ts_tablet_manager.cc:1428] T fbdc798725524341967c066f767a4506 P f5e6c5538fa84a4b90380eb5c2784a3b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260927 16:39:43.698213 10865 tablet.cc:2369] T fbdc798725524341967c066f767a4506 P f5e6c5538fa84a4b90380eb5c2784a3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:39:43.849721 11156 raft_consensus.cc:482] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:39:43.849957 11156 raft_consensus.cc:504] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8582af6bbbd4a589a1fabcfa16b34b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36515 } } peers { permanent_uuid: "f5e6c5538fa84a4b90380eb5c2784a3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44207 } } peers { permanent_uuid: "8db9826a2651482095e9921f3d703990" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42847 } }
I20260927 16:39:43.851158 11156 leader_election.cc:288] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8582af6bbbd4a589a1fabcfa16b34b9 (127.0.98.3:36515), f5e6c5538fa84a4b90380eb5c2784a3b (127.0.98.1:44207)
W20260927 16:39:43.854797 11133 tablet.cc:2369] T fbdc798725524341967c066f767a4506 P f8582af6bbbd4a589a1fabcfa16b34b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:39:43.855695 11000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:43.851441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44207 (local address 127.0.98.2:35729)
0927 16:39:43.851578 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:43.851594 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:43.851692 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:43.851971 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:43.851974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:43.851994 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:43.852120 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:43.852129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.853124 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.853126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:43.854370 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:43.854380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.855180 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.855183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:43.855346 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:43.855389 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 16:39:43.855470 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 16:39:43.855538 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":2}
I20260927 16:39:43.855731 11146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:43.851541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35729 (local address 127.0.98.1:44207)
0927 16:39:43.851687 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:43.851693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:43.851711 (+ 18us) server_negotiation.cc:407] Connection header received
0927 16:39:43.851789 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:43.851793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:43.851853 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:43.851942 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:43.852231 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.853025 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.854488 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.855060 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.855164 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:43.855267 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:43.855502 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 16:39:43.855601 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 16:39:43.855834 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:43.851581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38199 (local address 127.0.98.3:36515)
0927 16:39:43.851692 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:43.851697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:43.852078 (+ 381us) server_negotiation.cc:407] Connection header received
0927 16:39:43.852151 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:43.852153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:43.852220 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:43.852296 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:39:43.852511 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.853198 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.854488 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.855062 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.855164 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:43.855246 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:43.855637 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 16:39:43.855713 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 16:39:43.855842 11158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:43.851441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36515 (local address 127.0.98.2:38199)
0927 16:39:43.851979 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:43.851991 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:39:43.852074 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:39:43.852313 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:39:43.852315 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:39:43.852328 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:39:43.852410 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:43.852414 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.853288 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.853291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:43.854375 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:39:43.854380 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.855349 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.855352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:39:43.855495 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:43.855519 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 16:39:43.855597 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 16:39:43.855663 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":219,"thread_start_us":120,"threads_started":1}
I20260927 16:39:43.856405 10818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbdc798725524341967c066f767a4506" candidate_uuid: "8db9826a2651482095e9921f3d703990" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e6c5538fa84a4b90380eb5c2784a3b" is_pre_election: true
I20260927 16:39:43.856400 11086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbdc798725524341967c066f767a4506" candidate_uuid: "8db9826a2651482095e9921f3d703990" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8582af6bbbd4a589a1fabcfa16b34b9" is_pre_election: true
I20260927 16:39:43.856683 10818 raft_consensus.cc:2454] T fbdc798725524341967c066f767a4506 P f5e6c5538fa84a4b90380eb5c2784a3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8db9826a2651482095e9921f3d703990 in term 0.
I20260927 16:39:43.856702 11086 raft_consensus.cc:2454] T fbdc798725524341967c066f767a4506 P f8582af6bbbd4a589a1fabcfa16b34b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8db9826a2651482095e9921f3d703990 in term 0.
I20260927 16:39:43.857167 10887 leader_election.cc:302] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8db9826a2651482095e9921f3d703990, f5e6c5538fa84a4b90380eb5c2784a3b, f8582af6bbbd4a589a1fabcfa16b34b9; no voters:
I20260927 16:39:43.857437 11156 raft_consensus.cc:2781] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:39:43.857509 11156 raft_consensus.cc:482] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:39:43.857538 11156 raft_consensus.cc:3037] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:43.858520 11156 raft_consensus.cc:504] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8582af6bbbd4a589a1fabcfa16b34b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36515 } } peers { permanent_uuid: "f5e6c5538fa84a4b90380eb5c2784a3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44207 } } peers { permanent_uuid: "8db9826a2651482095e9921f3d703990" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42847 } }
I20260927 16:39:43.858868 11156 leader_election.cc:288] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990 [CANDIDATE]: Term 1 election: Requested vote from peers f8582af6bbbd4a589a1fabcfa16b34b9 (127.0.98.3:36515), f5e6c5538fa84a4b90380eb5c2784a3b (127.0.98.1:44207)
I20260927 16:39:43.859201 11086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbdc798725524341967c066f767a4506" candidate_uuid: "8db9826a2651482095e9921f3d703990" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8582af6bbbd4a589a1fabcfa16b34b9"
I20260927 16:39:43.859274 10818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbdc798725524341967c066f767a4506" candidate_uuid: "8db9826a2651482095e9921f3d703990" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e6c5538fa84a4b90380eb5c2784a3b"
I20260927 16:39:43.859370 11086 raft_consensus.cc:3037] T fbdc798725524341967c066f767a4506 P f8582af6bbbd4a589a1fabcfa16b34b9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:43.859395 10818 raft_consensus.cc:3037] T fbdc798725524341967c066f767a4506 P f5e6c5538fa84a4b90380eb5c2784a3b [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:39:43.860353 10818 raft_consensus.cc:2454] T fbdc798725524341967c066f767a4506 P f5e6c5538fa84a4b90380eb5c2784a3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8db9826a2651482095e9921f3d703990 in term 1.
I20260927 16:39:43.860361 11086 raft_consensus.cc:2454] T fbdc798725524341967c066f767a4506 P f8582af6bbbd4a589a1fabcfa16b34b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8db9826a2651482095e9921f3d703990 in term 1.
I20260927 16:39:43.860606 10887 leader_election.cc:302] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8db9826a2651482095e9921f3d703990, f8582af6bbbd4a589a1fabcfa16b34b9; no voters:
I20260927 16:39:43.860781 11156 raft_consensus.cc:2781] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:39:43.860985 11156 raft_consensus.cc:686] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990 [term 1 LEADER]: Becoming Leader. State: Replica: 8db9826a2651482095e9921f3d703990, State: Running, Role: LEADER
I20260927 16:39:43.861294 11156 consensus_queue.cc:237] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8582af6bbbd4a589a1fabcfa16b34b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36515 } } peers { permanent_uuid: "f5e6c5538fa84a4b90380eb5c2784a3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44207 } } peers { permanent_uuid: "8db9826a2651482095e9921f3d703990" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42847 } }
I20260927 16:39:43.864230 10673 catalog_manager.cc:5495] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8db9826a2651482095e9921f3d703990 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8db9826a2651482095e9921f3d703990" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8582af6bbbd4a589a1fabcfa16b34b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5e6c5538fa84a4b90380eb5c2784a3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8db9826a2651482095e9921f3d703990" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42847 } health_report { overall_health: HEALTHY } } }
I20260927 16:39:43.883301 11148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:43.880296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40468 (local address 127.0.98.2:42847)
0927 16:39:43.880400 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 16:39:43.880403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:39:43.880415 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:39:43.880555 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:39:43.880556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:39:43.880604 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:39:43.880685 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:39:43.880887 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.881345 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.882095 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:39:43.882281 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:39:43.882348 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:39:43.882582 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:39:43.882962 (+ 380us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:39:43.883047 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 16:39:43.883105 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 16:39:43.883144 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 16:39:43.888324 11086 raft_consensus.cc:1261] T fbdc798725524341967c066f767a4506 P f8582af6bbbd4a589a1fabcfa16b34b9 [term 1 FOLLOWER]: Refusing update from remote peer 8db9826a2651482095e9921f3d703990: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:39:43.888363 10818 raft_consensus.cc:1261] T fbdc798725524341967c066f767a4506 P f5e6c5538fa84a4b90380eb5c2784a3b [term 1 FOLLOWER]: Refusing update from remote peer 8db9826a2651482095e9921f3d703990: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:39:43.888953 11159 consensus_queue.cc:1035] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5e6c5538fa84a4b90380eb5c2784a3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:39:43.889128 11156 consensus_queue.cc:1035] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8582af6bbbd4a589a1fabcfa16b34b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:39:43.898175 11163 mvcc.cc:205] Tried to move back new op lower bound from 7333999345200517120 to 7333999345098326016. Current Snapshot: MvccSnapshot[applied={T|T < 7333999345200517120}]
I20260927 16:39:43.903491 10671 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:39:43.903648 10671 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:39:43.905956 10671 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d1ac4e9c11548df87cec6ce59d441d9: Sending DeleteTablet for 3 replicas of tablet fbdc798725524341967c066f767a4506
I20260927 16:39:43.906677 11066 tablet_service.cc:1514] Processing DeleteTablet for tablet fbdc798725524341967c066f767a4506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:39:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37956
I20260927 16:39:43.906702 10798 tablet_service.cc:1514] Processing DeleteTablet for tablet fbdc798725524341967c066f767a4506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:39:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47538
I20260927 16:39:43.907135 10932 tablet_service.cc:1514] Processing DeleteTablet for tablet fbdc798725524341967c066f767a4506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:39:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40466
I20260927 16:39:43.907382 11173 tablet_replica.cc:330] T fbdc798725524341967c066f767a4506 P f5e6c5538fa84a4b90380eb5c2784a3b: stopping tablet replica
I20260927 16:39:43.907586 11174 tablet_replica.cc:330] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990: stopping tablet replica
I20260927 16:39:43.907721 11173 raft_consensus.cc:2229] T fbdc798725524341967c066f767a4506 P f5e6c5538fa84a4b90380eb5c2784a3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:39:43.907919 11174 raft_consensus.cc:2229] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:39:43.907958 11172 tablet_replica.cc:330] T fbdc798725524341967c066f767a4506 P f8582af6bbbd4a589a1fabcfa16b34b9: stopping tablet replica
I20260927 16:39:43.908162 11172 raft_consensus.cc:2229] T fbdc798725524341967c066f767a4506 P f8582af6bbbd4a589a1fabcfa16b34b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:39:43.908237 11174 raft_consensus.cc:2258] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:39:43.908284 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 10735
I20260927 16:39:43.908852 11172 raft_consensus.cc:2258] T fbdc798725524341967c066f767a4506 P f8582af6bbbd4a589a1fabcfa16b34b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:39:43.910104 11174 ts_tablet_manager.cc:1905] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:39:43.910233 11172 ts_tablet_manager.cc:1905] T fbdc798725524341967c066f767a4506 P f8582af6bbbd4a589a1fabcfa16b34b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:39:43.912571 11174 ts_tablet_manager.cc:1918] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:39:43.912571 11172 ts_tablet_manager.cc:1918] T fbdc798725524341967c066f767a4506 P f8582af6bbbd4a589a1fabcfa16b34b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:39:43.912694 11172 log.cc:1196] T fbdc798725524341967c066f767a4506 P f8582af6bbbd4a589a1fabcfa16b34b9: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-2/wal/wals/fbdc798725524341967c066f767a4506
I20260927 16:39:43.912782 11174 log.cc:1196] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790527121515808-392-0/minicluster-data/ts-1/wal/wals/fbdc798725524341967c066f767a4506
I20260927 16:39:43.913136 11172 ts_tablet_manager.cc:1939] T fbdc798725524341967c066f767a4506 P f8582af6bbbd4a589a1fabcfa16b34b9: Deleting consensus metadata
I20260927 16:39:43.913187 11174 ts_tablet_manager.cc:1939] T fbdc798725524341967c066f767a4506 P 8db9826a2651482095e9921f3d703990: Deleting consensus metadata
I20260927 16:39:43.914253 10657 catalog_manager.cc:4841] TS 8db9826a2651482095e9921f3d703990 (127.0.98.2:42847): tablet fbdc798725524341967c066f767a4506 (table test-table [id=d3340e94a29d4681858f12115249d8ac]) successfully deleted
I20260927 16:39:43.914253 10659 catalog_manager.cc:4841] TS f8582af6bbbd4a589a1fabcfa16b34b9 (127.0.98.3:36515): tablet fbdc798725524341967c066f767a4506 (table test-table [id=d3340e94a29d4681858f12115249d8ac]) successfully deleted
W20260927 16:39:43.917150 10660 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44207 (error 108)
I20260927 16:39:43.917925 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 10869
I20260927 16:39:43.918141 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:39:43.917867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44207 (local address 127.0.0.1:47544)
0927 16:39:43.918018 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 16:39:43.918075 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 16:39:43.918737 10660 catalog_manager.cc:4584] TS f5e6c5538fa84a4b90380eb5c2784a3b (127.0.98.1:44207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fbdc798725524341967c066f767a4506: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44207: connect: Connection refused (error 111)
I20260927 16:39:43.923563 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 11003
I20260927 16:39:43.928753 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 10641
2026-09-27T16:39:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2272 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:39:43 dist-test-slave-z40b krb5kdc[11178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:39:43 dist-test-slave-z40b krb5kdc[11178](info): set up 2 sockets
Sep 27 16:39:43 dist-test-slave-z40b krb5kdc[11178](info): commencing operation
krb5kdc: starting...
W20260927 16:39:44.944931 392 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:39:45 dist-test-slave-z40b krb5kdc[11178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:39:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:39:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 16:39:45.101969 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskmlIKrs/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/postgres -l logfile start
2026-09-27 16:39:47.430 UTC [11208] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 16:39:47.430 UTC [11208] LOG: listening on IPv4 address "127.0.98.20", port 39351
2026-09-27 16:39:47.433 UTC [11208] LOG: listening on Unix socket "/tmp/.s.PGSQL.39351"
2026-09-27 16:39:47.454 UTC [11211] LOG: database system was shut down at 2026-09-27 16:39:46 UTC
2026-09-27 16:39:47.457 UTC [11208] LOG: database system is ready to accept connections
I20260927 16:39:48.399927 392 mini_postgres.cc:96] Postgres bound to 39351
2026-09-27 16:39:48.406 UTC [11219] FATAL: database "slave" does not exist
127.0.98.20:39351 - accepting connections
I20260927 16:39:48.406757 392 mini_ranger.cc:161] Starting Ranger...
I20260927 16:39:48.421146 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 16:39:48.866643 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 16:39:48.866742 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ranger-admin
2026-09-27 16:39:49,140 [I] DB FLAVOR :POSTGRES
2026-09-27 16:39:49,141 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 16:39:49,141 [I] Checking connection..
2026-09-27 16:39:49,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 16:39:49,378 [I] Checking connection passed.
2026-09-27 16:39:49,379 [I] --------- Verifying version history table ---------
2026-09-27 16:39:49,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 16:39:49,617 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 16:39:49,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 16:39:49,844 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 16:39:49,844 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 16:39:49,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39351/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 16:39:50.087 UTC [11283] WARNING: there is no transaction in progress
2026-09-27 16:39:50,092 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 16:39:50,093 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 16:39:50,338 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 16:39:50,338 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 16:39:50,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 16:39:50,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 16:39:50,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-z40b.c.gcp-upstream.internal','N') ;"
2026-09-27 16:39:51,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 16:39:51,237 [I] Table x_portal_user does not exist in database ranger
2026-09-27 16:39:51,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 16:39:51,481 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 16:39:51,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 16:39:51,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 16:39:51,940 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 16:39:51,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39351/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 16:39:52.833 UTC [11391] WARNING: there is no transaction in progress
2026-09-27 16:39:52.849 UTC [11391] WARNING: there is no transaction in progress
2026-09-27 16:39:52.864 UTC [11391] WARNING: there is no transaction in progress
2026-09-27 16:39:52.878 UTC [11391] WARNING: there is no transaction in progress
2026-09-27 16:39:52.895 UTC [11391] WARNING: there is no transaction in progress
2026-09-27 16:39:52.914 UTC [11391] WARNING: there is no transaction in progress
2026-09-27 16:39:52.974 UTC [11391] WARNING: there is no transaction in progress
2026-09-27 16:39:52.979 UTC [11391] WARNING: there is no transaction in progress
2026-09-27 16:39:52.992 UTC [11391] WARNING: there is no transaction in progress
2026-09-27 16:39:53.001 UTC [11391] WARNING: there is no transaction in progress
2026-09-27 16:39:53.010 UTC [11391] WARNING: there is no transaction in progress
2026-09-27 16:39:53.022 UTC [11391] WARNING: there is no transaction in progress
2026-09-27 16:39:53.031 UTC [11391] WARNING: there is no transaction in progress
2026-09-27 16:39:53.037 UTC [11391] WARNING: there is no transaction in progress
2026-09-27 16:39:53.042 UTC [11391] WARNING: there is no transaction in progress
2026-09-27 16:39:53.048 UTC [11391] WARNING: there is no transaction in progress
2026-09-27 16:39:53,616 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 16:39:53,616 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 16:39:53,854 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 16:39:53,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 16:39:54,088 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 16:39:54,089 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 16:39:54,299 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 16:39:54,299 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 16:39:54,513 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 16:39:54,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-z40b.c.gcp-upstream.internal';"
2026-09-27 16:39:54,953 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 16:39:54,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39351/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 16:39:55,165 [I] DB_PATCHES have already been applied
I20260927 16:39:55.170419 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskmlIKrs/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 16:39:55.170499 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 16:39:55.173663 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:39131
Sep 27, 2026 4:39:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 4:39:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 4:39:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 4:39:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 4:39:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 4:39:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 4:39:55 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 16:39:56.219475 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.813s user 0.000s sys 0.009s
Sep 27, 2026 4:40:15 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 4:40:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 4:40:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 4:40:15 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 4:40:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 4:40:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 4:40:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 4:40:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 4:40:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 4:40:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 4:40:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 4:40:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 4:40:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 4:40:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 4:40:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 4:40:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 4:40:15 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 4:40:16 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 16:40:17.127977 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 16:40:17 dist-test-slave-z40b krb5kdc[11178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527217, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 16:40:17.212491 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 16:40:17.221239 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 16:40:18.017489 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 16:40:18.017587 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ranger-kms
I20260927 16:40:18.017700 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:39351
2026-09-27 16:40:18,195 [I] DB FLAVOR :POSTGRES
2026-09-27 16:40:18,196 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 16:40:18,196 [I] Checking connection
2026-09-27 16:40:18,196 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39351/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 16:40:18,447 [I] connection success
2026-09-27 16:40:18,447 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 16:40:18,447 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 16:40:18,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39351/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 16:40:18,660 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 16:40:18,660 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 16:40:18,660 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 16:40:18,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39351/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 16:40:18,912 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 16:40:19.354756 392 mini_ranger_kms.cc:310] Created kms service
I20260927 16:40:19.538837 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 16:40:19.579063 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 16:40:19 dist-test-slave-z40b krb5kdc[11178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527217, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:40:19.934136 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 16:40:19.934297 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 16:40:19.934347 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 16:40:19.937891 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11571
I20260927 16:40:19.938022 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:54515
Sep 27, 2026 4:40:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 4:40:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 4:40:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 4:40:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 4:40:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 4:40:21 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 16:40:22.075022 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.863s user 0.005s sys 0.158s
I20260927 16:40:22.075160 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 16:40:22.075297 392 mini_ranger_kms.cc:401] 127.0.98.20:54515/kms/v1/keys
Sep 27 16:40:22 dist-test-slave-z40b krb5kdc[11178](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790527222, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 16:40:22 dist-test-slave-z40b krb5kdc[11178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527222, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 16:40:22 dist-test-slave-z40b krb5kdc[11178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527222, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 16:40:24 dist-test-slave-z40b krb5kdc[11178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:40:25.028066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36111
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36111
--ranger_config_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54515/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:40:25.131409 11622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:40:25.131618 11622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:40:25.131650 11622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:40:25.135437 11622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:40:25.135488 11622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:40:25.135506 11622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:40:25.135521 11622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:40:25.135567 11622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:40:25.140578 11622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:54515/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36111
--ranger_config_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:40:25.141515 11622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:40:25.142441 11622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:40:25.151597 11630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:40:25.151655 11628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:40:25.151711 11627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:40:25.151829 11622 server_base.cc:1034] running on GCE node
I20260927 16:40:25.152354 11622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:40:25.153352 11622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:40:25.154521 11622 hybrid_clock.cc:648] HybridClock initialized: now 1790527225154503 us; error 35 us; skew 500 ppm
Sep 27 16:40:25 dist-test-slave-z40b krb5kdc[11178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:40:25.161908 11622 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:40:25.162793 11622 webserver.cc:460] Webserver started at http://127.0.98.62:36923/ using document root <none> and password file <none>
I20260927 16:40:25.163358 11622 fs_manager.cc:359] Metadata directory not provided
I20260927 16:40:25.163431 11622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:40:25.163620 11622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 16:40:25 dist-test-slave-z40b krb5kdc[11178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:40:25.212975 11622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "93c3aeb1d8724a8eb65523d059bb50ad"
format_stamp: "Formatted at 2026-09-27 16:40:25 on dist-test-slave-z40b"
server_key: "77bd677c37a3731bfa4003219d5e6195"
server_key_iv: "62169ee20673c3fc1897b135f6d7213c"
server_key_version: "kuduclusterkey@0"
I20260927 16:40:25.213688 11622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "93c3aeb1d8724a8eb65523d059bb50ad"
format_stamp: "Formatted at 2026-09-27 16:40:25 on dist-test-slave-z40b"
server_key: "77bd677c37a3731bfa4003219d5e6195"
server_key_iv: "62169ee20673c3fc1897b135f6d7213c"
server_key_version: "kuduclusterkey@0"
I20260927 16:40:25.218302 11622 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 16:40:25.233480 11637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:40:25.234942 11622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 16:40:25.235064 11622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "93c3aeb1d8724a8eb65523d059bb50ad"
format_stamp: "Formatted at 2026-09-27 16:40:25 on dist-test-slave-z40b"
server_key: "77bd677c37a3731bfa4003219d5e6195"
server_key_iv: "62169ee20673c3fc1897b135f6d7213c"
server_key_version: "kuduclusterkey@0"
I20260927 16:40:25.235183 11622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:40:25.268003 11622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:40:25.271265 11622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:40:25.271552 11622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:40:25.280501 11622 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36111
I20260927 16:40:25.280550 11689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36111 every 8 connection(s)
I20260927 16:40:25.281562 11622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:40:25.284173 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 11622
I20260927 16:40:25.284308 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:40:25.284636 392 external_mini_cluster.cc:1428] Setting key 5d974d561d895931d06a290bb7744bbf
I20260927 16:40:25.284878 11690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:40:25.292124 11690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93c3aeb1d8724a8eb65523d059bb50ad: Bootstrap starting.
I20260927 16:40:25.295321 11690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93c3aeb1d8724a8eb65523d059bb50ad: Neither blocks nor log segments found. Creating new log.
Sep 27 16:40:25 dist-test-slave-z40b krb5kdc[11178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:40:25.296111 11690 log.cc:824] T 00000000000000000000000000000000 P 93c3aeb1d8724a8eb65523d059bb50ad: Log is configured to *not* fsync() on all Append() calls
I20260927 16:40:25.298663 11690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93c3aeb1d8724a8eb65523d059bb50ad: No bootstrap required, opened a new log
I20260927 16:40:25.302467 11693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:40:25.286113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37534 (local address 127.0.98.62:36111)
0927 16:40:25.286594 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0927 16:40:25.286613 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 16:40:25.286656 (+ 43us) server_negotiation.cc:407] Connection header received
0927 16:40:25.287572 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:40:25.287596 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:40:25.288033 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:40:25.288368 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:40:25.289232 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:40:25.290737 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:40:25.292081 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:40:25.292755 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:40:25.292921 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:40:25.296227 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:40:25.296234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:40:25.296252 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:40:25.296281 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:40:25.299188 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:40:25.299676 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:40:25.299681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:40:25.299687 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:40:25.299818 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:40:25.300170 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:40:25.300173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:40:25.300175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:40:25.300650 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:40:25.300863 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 16:40:25.301139 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 16:40:25.301412 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":159,"threads_started":1}
I20260927 16:40:25.302695 11690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93c3aeb1d8724a8eb65523d059bb50ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c3aeb1d8724a8eb65523d059bb50ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36111 } }
I20260927 16:40:25.302953 11690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93c3aeb1d8724a8eb65523d059bb50ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:40:25.303045 11690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93c3aeb1d8724a8eb65523d059bb50ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93c3aeb1d8724a8eb65523d059bb50ad, State: Initialized, Role: FOLLOWER
I20260927 16:40:25.303508 11690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93c3aeb1d8724a8eb65523d059bb50ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c3aeb1d8724a8eb65523d059bb50ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36111 } }
I20260927 16:40:25.303687 11690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93c3aeb1d8724a8eb65523d059bb50ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:40:25.303759 11690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93c3aeb1d8724a8eb65523d059bb50ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:40:25.303850 11690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93c3aeb1d8724a8eb65523d059bb50ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:40:25.304713 11690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93c3aeb1d8724a8eb65523d059bb50ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c3aeb1d8724a8eb65523d059bb50ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36111 } }
I20260927 16:40:25.305032 11690 leader_election.cc:302] T 00000000000000000000000000000000 P 93c3aeb1d8724a8eb65523d059bb50ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93c3aeb1d8724a8eb65523d059bb50ad; no voters:
I20260927 16:40:25.305306 11690 leader_election.cc:288] T 00000000000000000000000000000000 P 93c3aeb1d8724a8eb65523d059bb50ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:40:25.305394 11695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93c3aeb1d8724a8eb65523d059bb50ad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:40:25.305598 11695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93c3aeb1d8724a8eb65523d059bb50ad [term 1 LEADER]: Becoming Leader. State: Replica: 93c3aeb1d8724a8eb65523d059bb50ad, State: Running, Role: LEADER
I20260927 16:40:25.305949 11695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93c3aeb1d8724a8eb65523d059bb50ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c3aeb1d8724a8eb65523d059bb50ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36111 } }
I20260927 16:40:25.306486 11690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93c3aeb1d8724a8eb65523d059bb50ad [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:40:25.307332 11695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93c3aeb1d8724a8eb65523d059bb50ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93c3aeb1d8724a8eb65523d059bb50ad. Latest consensus state: current_term: 1 leader_uuid: "93c3aeb1d8724a8eb65523d059bb50ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c3aeb1d8724a8eb65523d059bb50ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36111 } } }
I20260927 16:40:25.307477 11695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93c3aeb1d8724a8eb65523d059bb50ad [sys.catalog]: This master's current role is: LEADER
I20260927 16:40:25.307698 11696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93c3aeb1d8724a8eb65523d059bb50ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93c3aeb1d8724a8eb65523d059bb50ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c3aeb1d8724a8eb65523d059bb50ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36111 } } }
I20260927 16:40:25.307806 11696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93c3aeb1d8724a8eb65523d059bb50ad [sys.catalog]: This master's current role is: LEADER
I20260927 16:40:25.309453 11690 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 16:40:25.310773 11699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:40:25.316571 11699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:40:25.323822 11699 catalog_manager.cc:1347] Generated new cluster ID: 7b206b24080d466cba3151c5d4b6b637
I20260927 16:40:25.323885 11699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:40:25.346467 11699 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:40:25.348500 11699 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:40:25.354717 11699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93c3aeb1d8724a8eb65523d059bb50ad: Generated new TSK 0
I20260927 16:40:25.355603 11699 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:40:26.008353 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54515/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36111
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:40:26.143831 11735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:40:26.144069 11735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:40:26.144133 11735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:40:26.148080 11735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:40:26.148139 11735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:40:26.148214 11735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:40:26.153872 11735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:54515/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:40:26.155004 11735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:40:26.156402 11735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:40:26.166280 11743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:40:26.166318 11735 server_base.cc:1034] running on GCE node
W20260927 16:40:26.166543 11745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:40:26.166286 11742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:40:26.166996 11735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:40:26.167634 11735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:40:26.168864 11735 hybrid_clock.cc:648] HybridClock initialized: now 1790527226168824 us; error 54 us; skew 500 ppm
Sep 27 16:40:26 dist-test-slave-z40b krb5kdc[11178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:40:26.172060 11735 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:40:26.173046 11735 webserver.cc:460] Webserver started at http://127.0.98.1:33997/ using document root <none> and password file <none>
I20260927 16:40:26.173758 11735 fs_manager.cc:359] Metadata directory not provided
I20260927 16:40:26.173848 11735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:40:26.174144 11735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 16:40:26 dist-test-slave-z40b krb5kdc[11178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:40:26.198547 11735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-0/data/instance:
uuid: "c31f41646fab4fbe86444d408879c9d0"
format_stamp: "Formatted at 2026-09-27 16:40:26 on dist-test-slave-z40b"
server_key: "0eed43b6ae5b6cac3653c00b43149069"
server_key_iv: "dee1ac67f8a5fa4cb6385e585dad8ee1"
server_key_version: "kuduclusterkey@0"
I20260927 16:40:26.199242 11735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c31f41646fab4fbe86444d408879c9d0"
format_stamp: "Formatted at 2026-09-27 16:40:26 on dist-test-slave-z40b"
server_key: "0eed43b6ae5b6cac3653c00b43149069"
server_key_iv: "dee1ac67f8a5fa4cb6385e585dad8ee1"
server_key_version: "kuduclusterkey@0"
I20260927 16:40:26.205899 11735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 16:40:26.241328 11752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:40:26.248463 11735 fs_manager.cc:725] Time spent opening block manager: real 0.024s user 0.002s sys 0.000s
I20260927 16:40:26.248607 11735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "c31f41646fab4fbe86444d408879c9d0"
format_stamp: "Formatted at 2026-09-27 16:40:26 on dist-test-slave-z40b"
server_key: "0eed43b6ae5b6cac3653c00b43149069"
server_key_iv: "dee1ac67f8a5fa4cb6385e585dad8ee1"
server_key_version: "kuduclusterkey@0"
I20260927 16:40:26.248747 11735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:40:26.273972 11735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:40:26.277385 11735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:40:26.277613 11735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:40:26.278254 11735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:40:26.279241 11735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:40:26.279335 11735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:40:26.279460 11735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:40:26.279532 11735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:40:26.290979 11735 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40403
I20260927 16:40:26.292186 11735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:40:26.293725 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 11735
I20260927 16:40:26.293839 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:40:26.294142 392 external_mini_cluster.cc:1428] Setting key 24c7699c847146861c79ea21693eba43
I20260927 16:40:26.293241 11867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40403 every 8 connection(s)
Sep 27 16:40:26 dist-test-slave-z40b krb5kdc[11178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:40:26.315274 11872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:40:26.298199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34049 (local address 127.0.98.62:36111)
0927 16:40:26.300228 (+ 2029us) server_negotiation.cc:206] Beginning negotiation
0927 16:40:26.300236 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 16:40:26.300248 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:40:26.300286 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:40:26.300289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:40:26.300333 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:40:26.300441 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:40:26.301772 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:40:26.302611 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:40:26.307412 (+ 4801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:40:26.307978 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:40:26.308074 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:40:26.311157 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:40:26.311161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:40:26.311163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:40:26.311186 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:40:26.312969 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:40:26.313550 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:40:26.313554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:40:26.313555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:40:26.313643 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:40:26.314025 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:40:26.314027 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:40:26.314029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:40:26.314221 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:40:26.314305 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 16:40:26.314918 (+ 613us) server_negotiation.cc:299] Negotiation successful
0927 16:40:26.315030 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1961,"thread_start_us":96,"threads_started":1}
I20260927 16:40:26.316504 11871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:40:26.297183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36111 (local address 127.0.98.1:34049)
0927 16:40:26.298845 (+ 1662us) negotiation.cc:107] Waiting for socket to connect
0927 16:40:26.298896 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 16:40:26.299846 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:40:26.300704 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:40:26.300713 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:40:26.301019 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:40:26.301623 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:40:26.301639 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:40:26.305410 (+ 3771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:40:26.305413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:40:26.307269 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:40:26.307280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:40:26.308234 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:40:26.308238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:40:26.308444 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:40:26.309490 (+ 1046us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:40:26.309508 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:40:26.310988 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:40:26.313100 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:40:26.313105 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:40:26.313119 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:40:26.313435 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:40:26.313743 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:40:26.313746 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:40:26.313748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:40:26.313921 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:40:26.314353 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:40:26.314359 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:40:26.314659 (+ 300us) client_negotiation.cc:769] Sending connection context
0927 16:40:26.315511 (+ 852us) client_negotiation.cc:240] Negotiation successful
0927 16:40:26.315780 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1465,"thread_start_us":140,"threads_started":1}
I20260927 16:40:26.317673 11868 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36111
I20260927 16:40:26.317885 11868 heartbeater.cc:464] Registering TS with master...
I20260927 16:40:26.318322 11868 heartbeater.cc:511] Master 127.0.98.62:36111 requested a full tablet report, sending...
I20260927 16:40:26.319772 11653 ts_manager.cc:194] Registered new tserver with Master: c31f41646fab4fbe86444d408879c9d0 (127.0.98.1:40403)
I20260927 16:40:26.320814 11653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34049
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:40:26.343626 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54515/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36111
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 16:40:26 dist-test-slave-z40b krb5kdc[11178](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790527226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 16:40:26.495936 11874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:40:26.496304 11874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:40:26.496439 11874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:40:26.502707 11874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:40:26.502833 11874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:40:26.502948 11874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:40:26.508502 11874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:54515/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:40:26.509814 11874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:40:26.510990 11874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:40:26.520597 11881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:40:26.521008 11883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:40:26.521222 11880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:40:26.522382 11874 server_base.cc:1034] running on GCE node
I20260927 16:40:26.522799 11874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:40:26.523507 11874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:40:26.524667 11874 hybrid_clock.cc:648] HybridClock initialized: now 1790527226524638 us; error 40 us; skew 500 ppm
Sep 27 16:40:26 dist-test-slave-z40b krb5kdc[11178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:40:26.528152 11874 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:40:26.529091 11874 webserver.cc:460] Webserver started at http://127.0.98.2:46199/ using document root <none> and password file <none>
I20260927 16:40:26.529752 11874 fs_manager.cc:359] Metadata directory not provided
I20260927 16:40:26.529883 11874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:40:26.530120 11874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 16:40:26 dist-test-slave-z40b krb5kdc[11178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:40:26.542982 11874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-1/data/instance:
uuid: "7395ddd784a545f191d8957cbd45be31"
format_stamp: "Formatted at 2026-09-27 16:40:26 on dist-test-slave-z40b"
server_key: "919e2cc8381d43e60debe847165891ef"
server_key_iv: "f7777bab901568e6eab483101bbacab7"
server_key_version: "kuduclusterkey@0"
I20260927 16:40:26.543555 11874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7395ddd784a545f191d8957cbd45be31"
format_stamp: "Formatted at 2026-09-27 16:40:26 on dist-test-slave-z40b"
server_key: "919e2cc8381d43e60debe847165891ef"
server_key_iv: "f7777bab901568e6eab483101bbacab7"
server_key_version: "kuduclusterkey@0"
I20260927 16:40:26.547195 11874 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:40:26.567776 11890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:40:26.569350 11874 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260927 16:40:26.569517 11874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "7395ddd784a545f191d8957cbd45be31"
format_stamp: "Formatted at 2026-09-27 16:40:26 on dist-test-slave-z40b"
server_key: "919e2cc8381d43e60debe847165891ef"
server_key_iv: "f7777bab901568e6eab483101bbacab7"
server_key_version: "kuduclusterkey@0"
I20260927 16:40:26.569707 11874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:40:26.590893 11874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:40:26.595449 11874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:40:26.595690 11874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:40:26.596376 11874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:40:26.597473 11874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:40:26.597579 11874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:40:26.597687 11874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:40:26.597752 11874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:40:26.611768 11874 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41555
I20260927 16:40:26.612952 11874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:40:26.615756 12003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41555 every 8 connection(s)
I20260927 16:40:26.622987 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 11874
I20260927 16:40:26.623106 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:40:26.623497 392 external_mini_cluster.cc:1428] Setting key bbb406e2123769cc27c1c26d3c72bbc5
Sep 27 16:40:26 dist-test-slave-z40b krb5kdc[11178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:40:26.633534 11872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:40:26.619404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35317 (local address 127.0.98.62:36111)
0927 16:40:26.619542 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 16:40:26.619547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:40:26.620454 (+ 907us) server_negotiation.cc:407] Connection header received
0927 16:40:26.621752 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:40:26.621756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:40:26.621827 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:40:26.621900 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:40:26.623549 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:40:26.624362 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:40:26.626042 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:40:26.626377 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:40:26.626478 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:40:26.629394 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:40:26.629399 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:40:26.629401 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:40:26.629434 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:40:26.631168 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:40:26.631781 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:40:26.631785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:40:26.631787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:40:26.631874 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:40:26.632298 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:40:26.632302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:40:26.632304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:40:26.632514 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:40:26.632597 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 16:40:26.633254 (+ 657us) server_negotiation.cc:299] Negotiation successful
0927 16:40:26.633378 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:40:26.636520 12006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:40:26.618394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36111 (local address 127.0.98.2:35317)
0927 16:40:26.620293 (+ 1899us) negotiation.cc:107] Waiting for socket to connect
0927 16:40:26.620348 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 16:40:26.621430 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:40:26.622298 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:40:26.622306 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:40:26.622647 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:40:26.623374 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:40:26.623394 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:40:26.624529 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:40:26.624532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:40:26.625899 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:40:26.625910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:40:26.626673 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:40:26.626677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:40:26.626902 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:40:26.627564 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:40:26.627583 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:40:26.629210 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:40:26.631333 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:40:26.631339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:40:26.631352 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:40:26.631651 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:40:26.632007 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:40:26.632010 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:40:26.632012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:40:26.632182 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:40:26.632667 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:40:26.632672 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:40:26.632983 (+ 311us) client_negotiation.cc:769] Sending connection context
0927 16:40:26.633941 (+ 958us) client_negotiation.cc:240] Negotiation successful
0927 16:40:26.634298 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1693,"thread_start_us":137,"threads_started":1}
I20260927 16:40:26.638590 12004 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36111
I20260927 16:40:26.638805 12004 heartbeater.cc:464] Registering TS with master...
I20260927 16:40:26.639335 12004 heartbeater.cc:511] Master 127.0.98.62:36111 requested a full tablet report, sending...
I20260927 16:40:26.640434 11653 ts_manager.cc:194] Registered new tserver with Master: 7395ddd784a545f191d8957cbd45be31 (127.0.98.2:41555)
I20260927 16:40:26.641126 11653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35317
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:40:26.683766 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54515/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36111
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:40:26.892221 12009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:40:26.892620 12009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:40:26.892700 12009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:40:26.898937 12009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:40:26.899015 12009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
Sep 27 16:40:26 dist-test-slave-z40b krb5kdc[11178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 16:40:26.899899 12009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:40:26.908490 12009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:54515/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:40:26.909845 12009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:40:26.911137 12009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:40:26.938164 12015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:40:26.938684 12017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:40:26.938923 12014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:40:26.944446 12009 server_base.cc:1034] running on GCE node
I20260927 16:40:26.945021 12009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:40:26.945820 12009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:40:26.978462 12009 hybrid_clock.cc:648] HybridClock initialized: now 1790527226978359 us; error 118 us; skew 500 ppm
Sep 27 16:40:26 dist-test-slave-z40b krb5kdc[11178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:40:26.982812 12009 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:40:26.984009 12009 webserver.cc:460] Webserver started at http://127.0.98.3:35499/ using document root <none> and password file <none>
I20260927 16:40:26.984741 12009 fs_manager.cc:359] Metadata directory not provided
I20260927 16:40:26.984834 12009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:40:26.985301 12009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 16:40:26 dist-test-slave-z40b krb5kdc[11178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:40:27.003821 12009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-2/data/instance:
uuid: "1b05ed463c35461c9067db7784ca88a9"
format_stamp: "Formatted at 2026-09-27 16:40:26 on dist-test-slave-z40b"
server_key: "8bd09bdcd67406b292ddc1fd4cea5d99"
server_key_iv: "00939cffd179bbea15b0d45bbc9d21e3"
server_key_version: "kuduclusterkey@0"
I20260927 16:40:27.004570 12009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1b05ed463c35461c9067db7784ca88a9"
format_stamp: "Formatted at 2026-09-27 16:40:26 on dist-test-slave-z40b"
server_key: "8bd09bdcd67406b292ddc1fd4cea5d99"
server_key_iv: "00939cffd179bbea15b0d45bbc9d21e3"
server_key_version: "kuduclusterkey@0"
I20260927 16:40:27.015632 12009 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.004s sys 0.000s
I20260927 16:40:27.048825 12024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:40:27.050580 12009 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.003s sys 0.000s
I20260927 16:40:27.050738 12009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "1b05ed463c35461c9067db7784ca88a9"
format_stamp: "Formatted at 2026-09-27 16:40:26 on dist-test-slave-z40b"
server_key: "8bd09bdcd67406b292ddc1fd4cea5d99"
server_key_iv: "00939cffd179bbea15b0d45bbc9d21e3"
server_key_version: "kuduclusterkey@0"
I20260927 16:40:27.050871 12009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:40:27.065244 12009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:40:27.068866 12009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:40:27.069494 12009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:40:27.070252 12009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:40:27.071399 12009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:40:27.071509 12009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:40:27.071604 12009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:40:27.072053 12009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:40:27.083803 12009 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39189
I20260927 16:40:27.084664 12009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:40:27.085824 12137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39189 every 8 connection(s)
I20260927 16:40:27.089640 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 12009
I20260927 16:40:27.089759 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:40:27.090070 392 external_mini_cluster.cc:1428] Setting key a1fab1f6fc5e2c98b8f7ebd766c077b3
Sep 27 16:40:27 dist-test-slave-z40b krb5kdc[11178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:40:27.098155 11872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:40:27.086564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38927 (local address 127.0.98.62:36111)
0927 16:40:27.086713 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 16:40:27.086717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:40:27.087544 (+ 827us) server_negotiation.cc:407] Connection header received
0927 16:40:27.088655 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:40:27.088658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:40:27.088716 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:40:27.088767 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:40:27.090280 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.091016 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.092450 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.092649 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.092746 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:40:27.094831 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:40:27.094834 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:40:27.094837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:40:27.094860 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:40:27.096322 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:40:27.096794 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:40:27.096800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:40:27.096805 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:40:27.096882 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:40:27.097194 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:40:27.097197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:40:27.097198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:40:27.097367 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:40:27.097463 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 16:40:27.097902 (+ 439us) server_negotiation.cc:299] Negotiation successful
0927 16:40:27.098007 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 16:40:27.098909 12140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:40:27.086952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36111 (local address 127.0.98.3:38927)
0927 16:40:27.087400 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 16:40:27.087450 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 16:40:27.088395 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:40:27.089161 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:40:27.089172 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:40:27.089494 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:40:27.090122 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:40:27.090140 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.091160 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.091163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:40:27.092343 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:40:27.092351 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.092774 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.092777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:40:27.092941 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:40:27.093470 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:40:27.093485 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:40:27.094694 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:40:27.096430 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:40:27.096440 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:40:27.096449 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:40:27.096697 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:40:27.096967 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:40:27.096972 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:40:27.096974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:40:27.097096 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:40:27.097461 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:40:27.097467 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:40:27.097700 (+ 233us) client_negotiation.cc:769] Sending connection context
0927 16:40:27.097867 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 16:40:27.098119 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":123,"threads_started":1}
I20260927 16:40:27.100085 12138 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36111
I20260927 16:40:27.100272 12138 heartbeater.cc:464] Registering TS with master...
I20260927 16:40:27.100728 12138 heartbeater.cc:511] Master 127.0.98.62:36111 requested a full tablet report, sending...
I20260927 16:40:27.101694 11653 ts_manager.cc:194] Registered new tserver with Master: 1b05ed463c35461c9067db7784ca88a9 (127.0.98.3:39189)
I20260927 16:40:27.102373 11653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38927
I20260927 16:40:27.105629 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:40:27.120785 11872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:40:27.111622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49746 (local address 127.0.98.62:36111)
0927 16:40:27.111716 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 16:40:27.111720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:40:27.112299 (+ 579us) server_negotiation.cc:407] Connection header received
0927 16:40:27.112462 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:40:27.112465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:40:27.112503 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:40:27.112565 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:40:27.113481 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.114231 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.115375 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.115635 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.115723 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:40:27.117521 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:40:27.117525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:40:27.117527 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:40:27.117550 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:40:27.119178 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:40:27.119636 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:40:27.119639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:40:27.119641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:40:27.119712 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:40:27.119968 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:40:27.119971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:40:27.119973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:40:27.120141 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:40:27.120230 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 16:40:27.120562 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 16:40:27.120654 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 16:40:27.124135 11653 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:40:27.128149 11653 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:40:27.153998 12152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:40:27.138758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39189 (local address 127.0.0.1:45842)
0927 16:40:27.147728 (+ 8970us) negotiation.cc:107] Waiting for socket to connect
0927 16:40:27.147793 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 16:40:27.147907 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:40:27.148983 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:40:27.148986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:40:27.149001 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:40:27.149080 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:40:27.149084 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.150394 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.150398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:40:27.152292 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:40:27.152301 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.153482 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.153485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:40:27.153585 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:40:27.153619 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 16:40:27.153720 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 16:40:27.153832 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8811,"mutex_wait_us":804,"thread_start_us":82,"threads_started":1}
I20260927 16:40:27.154408 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:40:27.137595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.0.98.2:41555)
0927 16:40:27.147423 (+ 9828us) server_negotiation.cc:206] Beginning negotiation
0927 16:40:27.147439 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 16:40:27.148052 (+ 613us) server_negotiation.cc:407] Connection header received
0927 16:40:27.148134 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:40:27.148140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:40:27.148365 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:40:27.148522 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:40:27.149848 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.150916 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.152694 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.153400 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.153492 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:40:27.153676 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 16:40:27.154137 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 16:40:27.154251 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9704,"thread_start_us":121,"threads_started":1}
I20260927 16:40:27.154042 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:40:27.137463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41555 (local address 127.0.0.1:49350)
0927 16:40:27.147728 (+ 10265us) negotiation.cc:107] Waiting for socket to connect
0927 16:40:27.147787 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 16:40:27.147921 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:40:27.148723 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:40:27.148729 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:40:27.148763 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:40:27.149717 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:40:27.149724 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.151040 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.151043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:40:27.152580 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:40:27.152587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.153729 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.153732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:40:27.153817 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:40:27.153835 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 16:40:27.153893 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 16:40:27.153943 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10058,"mutex_wait_us":1305,"spinlock_wait_cycles":4864,"thread_start_us":79,"threads_started":1}
I20260927 16:40:27.154906 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:40:27.139415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.0.98.3:39189)
0927 16:40:27.148392 (+ 8977us) server_negotiation.cc:206] Beginning negotiation
0927 16:40:27.148402 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:40:27.148423 (+ 21us) server_negotiation.cc:407] Connection header received
0927 16:40:27.148490 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:40:27.148495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:40:27.148704 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:40:27.148846 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:40:27.149206 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.150268 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.152429 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.153178 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.153276 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:40:27.153422 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 16:40:27.154634 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0927 16:40:27.154741 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8877,"thread_start_us":98,"threads_started":1}
I20260927 16:40:27.158236 11938 tablet_service.cc:1467] Processing CreateTablet for tablet f2149cdb30e24ce593a6f073a4e61edb (DEFAULT_TABLE table=test-table [id=7a2d940fccf04d6da1e2e2ebf30d77bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:40:27.159471 11938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2149cdb30e24ce593a6f073a4e61edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:40:27.161950 12070 tablet_service.cc:1467] Processing CreateTablet for tablet f2149cdb30e24ce593a6f073a4e61edb (DEFAULT_TABLE table=test-table [id=7a2d940fccf04d6da1e2e2ebf30d77bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:40:27.162997 12070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2149cdb30e24ce593a6f073a4e61edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:40:27.169979 12153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:40:27.139023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40403 (local address 127.0.0.1:36280)
0927 16:40:27.148080 (+ 9057us) negotiation.cc:107] Waiting for socket to connect
0927 16:40:27.148091 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:40:27.148162 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:40:27.148723 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:40:27.148729 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:40:27.148764 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:40:27.148920 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:40:27.148928 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.150330 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.150333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:40:27.151700 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:40:27.151710 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.169468 (+ 17758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.169472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:40:27.169606 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:40:27.169648 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 16:40:27.169724 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 16:40:27.169828 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9009,"mutex_wait_us":4,"spinlock_wait_cycles":5120,"thread_start_us":67,"threads_started":1}
I20260927 16:40:27.170708 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:40:27.139195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36280 (local address 127.0.98.1:40403)
0927 16:40:27.143435 (+ 4240us) server_negotiation.cc:206] Beginning negotiation
0927 16:40:27.143443 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 16:40:27.148164 (+ 4721us) server_negotiation.cc:407] Connection header received
0927 16:40:27.148243 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:40:27.148247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:40:27.148396 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:40:27.148522 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:40:27.149071 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.150152 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.168216 (+ 18064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.169112 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.169206 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:40:27.169343 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 16:40:27.170266 (+ 923us) server_negotiation.cc:299] Negotiation successful
0927 16:40:27.170396 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4107,"thread_start_us":89,"threads_started":1}
I20260927 16:40:27.172200 12156 tablet_bootstrap.cc:492] T f2149cdb30e24ce593a6f073a4e61edb P 7395ddd784a545f191d8957cbd45be31: Bootstrap starting.
I20260927 16:40:27.174743 12157 tablet_bootstrap.cc:492] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9: Bootstrap starting.
I20260927 16:40:27.174368 11789 tablet_service.cc:1467] Processing CreateTablet for tablet f2149cdb30e24ce593a6f073a4e61edb (DEFAULT_TABLE table=test-table [id=7a2d940fccf04d6da1e2e2ebf30d77bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:40:27.175547 11789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2149cdb30e24ce593a6f073a4e61edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:40:27.177652 12157 tablet_bootstrap.cc:654] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9: Neither blocks nor log segments found. Creating new log.
I20260927 16:40:27.178313 12156 tablet_bootstrap.cc:654] T f2149cdb30e24ce593a6f073a4e61edb P 7395ddd784a545f191d8957cbd45be31: Neither blocks nor log segments found. Creating new log.
I20260927 16:40:27.179256 12156 log.cc:824] T f2149cdb30e24ce593a6f073a4e61edb P 7395ddd784a545f191d8957cbd45be31: Log is configured to *not* fsync() on all Append() calls
I20260927 16:40:27.180115 12157 log.cc:824] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9: Log is configured to *not* fsync() on all Append() calls
I20260927 16:40:27.182509 12157 tablet_bootstrap.cc:492] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9: No bootstrap required, opened a new log
I20260927 16:40:27.182701 12157 ts_tablet_manager.cc:1397] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 16:40:27.187137 12157 raft_consensus.cc:348] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7395ddd784a545f191d8957cbd45be31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41555 } } peers { permanent_uuid: "1b05ed463c35461c9067db7784ca88a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39189 } } peers { permanent_uuid: "c31f41646fab4fbe86444d408879c9d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40403 } }
I20260927 16:40:27.187613 12157 raft_consensus.cc:374] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:40:27.187721 12157 raft_consensus.cc:729] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b05ed463c35461c9067db7784ca88a9, State: Initialized, Role: FOLLOWER
I20260927 16:40:27.188267 12157 consensus_queue.cc:260] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7395ddd784a545f191d8957cbd45be31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41555 } } peers { permanent_uuid: "1b05ed463c35461c9067db7784ca88a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39189 } } peers { permanent_uuid: "c31f41646fab4fbe86444d408879c9d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40403 } }
I20260927 16:40:27.189308 12157 ts_tablet_manager.cc:1428] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 16:40:27.189692 12138 heartbeater.cc:503] Master 127.0.98.62:36111 was elected leader, sending a full tablet report...
I20260927 16:40:27.191594 12160 tablet_bootstrap.cc:492] T f2149cdb30e24ce593a6f073a4e61edb P c31f41646fab4fbe86444d408879c9d0: Bootstrap starting.
I20260927 16:40:27.193948 12160 tablet_bootstrap.cc:654] T f2149cdb30e24ce593a6f073a4e61edb P c31f41646fab4fbe86444d408879c9d0: Neither blocks nor log segments found. Creating new log.
I20260927 16:40:27.195410 12160 log.cc:824] T f2149cdb30e24ce593a6f073a4e61edb P c31f41646fab4fbe86444d408879c9d0: Log is configured to *not* fsync() on all Append() calls
I20260927 16:40:27.196081 12156 tablet_bootstrap.cc:492] T f2149cdb30e24ce593a6f073a4e61edb P 7395ddd784a545f191d8957cbd45be31: No bootstrap required, opened a new log
I20260927 16:40:27.196280 12156 ts_tablet_manager.cc:1397] T f2149cdb30e24ce593a6f073a4e61edb P 7395ddd784a545f191d8957cbd45be31: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.000s
I20260927 16:40:27.202690 12156 raft_consensus.cc:348] T f2149cdb30e24ce593a6f073a4e61edb P 7395ddd784a545f191d8957cbd45be31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7395ddd784a545f191d8957cbd45be31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41555 } } peers { permanent_uuid: "1b05ed463c35461c9067db7784ca88a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39189 } } peers { permanent_uuid: "c31f41646fab4fbe86444d408879c9d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40403 } }
I20260927 16:40:27.203025 12156 raft_consensus.cc:374] T f2149cdb30e24ce593a6f073a4e61edb P 7395ddd784a545f191d8957cbd45be31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:40:27.203120 12156 raft_consensus.cc:729] T f2149cdb30e24ce593a6f073a4e61edb P 7395ddd784a545f191d8957cbd45be31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7395ddd784a545f191d8957cbd45be31, State: Initialized, Role: FOLLOWER
I20260927 16:40:27.203665 12156 consensus_queue.cc:260] T f2149cdb30e24ce593a6f073a4e61edb P 7395ddd784a545f191d8957cbd45be31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7395ddd784a545f191d8957cbd45be31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41555 } } peers { permanent_uuid: "1b05ed463c35461c9067db7784ca88a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39189 } } peers { permanent_uuid: "c31f41646fab4fbe86444d408879c9d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40403 } }
I20260927 16:40:27.204721 12156 ts_tablet_manager.cc:1428] T f2149cdb30e24ce593a6f073a4e61edb P 7395ddd784a545f191d8957cbd45be31: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
I20260927 16:40:27.205084 12004 heartbeater.cc:503] Master 127.0.98.62:36111 was elected leader, sending a full tablet report...
I20260927 16:40:27.205103 12160 tablet_bootstrap.cc:492] T f2149cdb30e24ce593a6f073a4e61edb P c31f41646fab4fbe86444d408879c9d0: No bootstrap required, opened a new log
I20260927 16:40:27.205273 12160 ts_tablet_manager.cc:1397] T f2149cdb30e24ce593a6f073a4e61edb P c31f41646fab4fbe86444d408879c9d0: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20260927 16:40:27.207542 12161 raft_consensus.cc:482] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:40:27.207748 12161 raft_consensus.cc:504] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7395ddd784a545f191d8957cbd45be31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41555 } } peers { permanent_uuid: "1b05ed463c35461c9067db7784ca88a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39189 } } peers { permanent_uuid: "c31f41646fab4fbe86444d408879c9d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40403 } }
I20260927 16:40:27.209550 12160 raft_consensus.cc:348] T f2149cdb30e24ce593a6f073a4e61edb P c31f41646fab4fbe86444d408879c9d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7395ddd784a545f191d8957cbd45be31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41555 } } peers { permanent_uuid: "1b05ed463c35461c9067db7784ca88a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39189 } } peers { permanent_uuid: "c31f41646fab4fbe86444d408879c9d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40403 } }
I20260927 16:40:27.209862 12160 raft_consensus.cc:374] T f2149cdb30e24ce593a6f073a4e61edb P c31f41646fab4fbe86444d408879c9d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:40:27.209973 12160 raft_consensus.cc:729] T f2149cdb30e24ce593a6f073a4e61edb P c31f41646fab4fbe86444d408879c9d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c31f41646fab4fbe86444d408879c9d0, State: Initialized, Role: FOLLOWER
I20260927 16:40:27.210506 12160 consensus_queue.cc:260] T f2149cdb30e24ce593a6f073a4e61edb P c31f41646fab4fbe86444d408879c9d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7395ddd784a545f191d8957cbd45be31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41555 } } peers { permanent_uuid: "1b05ed463c35461c9067db7784ca88a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39189 } } peers { permanent_uuid: "c31f41646fab4fbe86444d408879c9d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40403 } }
I20260927 16:40:27.211575 12160 ts_tablet_manager.cc:1428] T f2149cdb30e24ce593a6f073a4e61edb P c31f41646fab4fbe86444d408879c9d0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 16:40:27.213812 12140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:40:27.209044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41555 (local address 127.0.98.3:51675)
0927 16:40:27.209291 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 16:40:27.209307 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:40:27.209400 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:40:27.209946 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:40:27.209950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:40:27.209970 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:40:27.210110 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:40:27.210117 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.211076 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.211079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:40:27.212438 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:40:27.212448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.213416 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.213420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:40:27.213524 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:40:27.213545 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 16:40:27.213614 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 16:40:27.213684 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260927 16:40:27.214250 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:40:27.209535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51675 (local address 127.0.98.2:41555)
0927 16:40:27.209654 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 16:40:27.209658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:40:27.209670 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:40:27.209712 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:40:27.209714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:40:27.209753 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:40:27.209824 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:40:27.210233 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.210954 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.212577 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.213182 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.213284 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:40:27.213346 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 16:40:27.214066 (+ 720us) server_negotiation.cc:299] Negotiation successful
0927 16:40:27.214136 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 16:40:27.214319 11868 heartbeater.cc:503] Master 127.0.98.62:36111 was elected leader, sending a full tablet report...
I20260927 16:40:27.214762 11943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2149cdb30e24ce593a6f073a4e61edb" candidate_uuid: "1b05ed463c35461c9067db7784ca88a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7395ddd784a545f191d8957cbd45be31" is_pre_election: true
I20260927 16:40:27.214998 11943 raft_consensus.cc:2454] T f2149cdb30e24ce593a6f073a4e61edb P 7395ddd784a545f191d8957cbd45be31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b05ed463c35461c9067db7784ca88a9 in term 0.
I20260927 16:40:27.215749 12028 leader_election.cc:302] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b05ed463c35461c9067db7784ca88a9, 7395ddd784a545f191d8957cbd45be31; no voters:
I20260927 16:40:27.220889 12140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:40:27.216365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40403 (local address 127.0.98.3:59835)
0927 16:40:27.216990 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0927 16:40:27.217005 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:40:27.217080 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:40:27.217413 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:40:27.217416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:40:27.217431 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:40:27.217501 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:40:27.217506 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.218432 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.218435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:40:27.219553 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:40:27.219564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.220537 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.220541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:40:27.220622 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:40:27.220647 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 16:40:27.220717 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 16:40:27.220777 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549}
I20260927 16:40:27.221302 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:40:27.216749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59835 (local address 127.0.98.1:40403)
0927 16:40:27.216853 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 16:40:27.216857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:40:27.217151 (+ 294us) server_negotiation.cc:407] Connection header received
0927 16:40:27.217193 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:40:27.217195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:40:27.217235 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:40:27.217295 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:40:27.217611 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.218303 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.219705 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.220324 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.220430 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:40:27.220484 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 16:40:27.221129 (+ 645us) server_negotiation.cc:299] Negotiation successful
0927 16:40:27.221202 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 16:40:27.222121 11822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2149cdb30e24ce593a6f073a4e61edb" candidate_uuid: "1b05ed463c35461c9067db7784ca88a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c31f41646fab4fbe86444d408879c9d0" is_pre_election: true
I20260927 16:40:27.222363 11822 raft_consensus.cc:2454] T f2149cdb30e24ce593a6f073a4e61edb P c31f41646fab4fbe86444d408879c9d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b05ed463c35461c9067db7784ca88a9 in term 0.
I20260927 16:40:27.222870 12161 leader_election.cc:288] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7395ddd784a545f191d8957cbd45be31 (127.0.98.2:41555), c31f41646fab4fbe86444d408879c9d0 (127.0.98.1:40403)
I20260927 16:40:27.223176 12161 raft_consensus.cc:2781] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:40:27.223315 12161 raft_consensus.cc:482] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:40:27.223425 12161 raft_consensus.cc:3037] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:40:27.224824 12161 raft_consensus.cc:504] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7395ddd784a545f191d8957cbd45be31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41555 } } peers { permanent_uuid: "1b05ed463c35461c9067db7784ca88a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39189 } } peers { permanent_uuid: "c31f41646fab4fbe86444d408879c9d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40403 } }
I20260927 16:40:27.226895 11943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2149cdb30e24ce593a6f073a4e61edb" candidate_uuid: "1b05ed463c35461c9067db7784ca88a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7395ddd784a545f191d8957cbd45be31"
I20260927 16:40:27.227012 11943 raft_consensus.cc:3037] T f2149cdb30e24ce593a6f073a4e61edb P 7395ddd784a545f191d8957cbd45be31 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:40:27.228238 11943 raft_consensus.cc:2454] T f2149cdb30e24ce593a6f073a4e61edb P 7395ddd784a545f191d8957cbd45be31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b05ed463c35461c9067db7784ca88a9 in term 1.
I20260927 16:40:27.228586 12028 leader_election.cc:302] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b05ed463c35461c9067db7784ca88a9, 7395ddd784a545f191d8957cbd45be31; no voters:
I20260927 16:40:27.228744 12163 raft_consensus.cc:2781] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:40:27.228960 12163 raft_consensus.cc:686] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9 [term 1 LEADER]: Becoming Leader. State: Replica: 1b05ed463c35461c9067db7784ca88a9, State: Running, Role: LEADER
I20260927 16:40:27.229132 11822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2149cdb30e24ce593a6f073a4e61edb" candidate_uuid: "1b05ed463c35461c9067db7784ca88a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c31f41646fab4fbe86444d408879c9d0"
I20260927 16:40:27.229225 11822 raft_consensus.cc:3037] T f2149cdb30e24ce593a6f073a4e61edb P c31f41646fab4fbe86444d408879c9d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:40:27.229290 12163 consensus_queue.cc:237] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7395ddd784a545f191d8957cbd45be31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41555 } } peers { permanent_uuid: "1b05ed463c35461c9067db7784ca88a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39189 } } peers { permanent_uuid: "c31f41646fab4fbe86444d408879c9d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40403 } }
I20260927 16:40:27.230383 11822 raft_consensus.cc:2454] T f2149cdb30e24ce593a6f073a4e61edb P c31f41646fab4fbe86444d408879c9d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b05ed463c35461c9067db7784ca88a9 in term 1.
I20260927 16:40:27.230667 12161 leader_election.cc:288] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9 [CANDIDATE]: Term 1 election: Requested vote from peers 7395ddd784a545f191d8957cbd45be31 (127.0.98.2:41555), c31f41646fab4fbe86444d408879c9d0 (127.0.98.1:40403)
I20260927 16:40:27.231984 11653 catalog_manager.cc:5495] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b05ed463c35461c9067db7784ca88a9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1b05ed463c35461c9067db7784ca88a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7395ddd784a545f191d8957cbd45be31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b05ed463c35461c9067db7784ca88a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c31f41646fab4fbe86444d408879c9d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40403 } health_report { overall_health: UNKNOWN } } }
I20260927 16:40:27.248450 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:40:27.240194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45848 (local address 127.0.98.3:39189)
0927 16:40:27.240310 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 16:40:27.240316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:40:27.240330 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:40:27.240405 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:40:27.240408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:40:27.240462 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:40:27.240551 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:40:27.240923 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.241649 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.242871 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:40:27.243165 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:40:27.246696 (+ 3531us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:40:27.247133 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:40:27.247694 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:40:27.247823 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 16:40:27.248245 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 16:40:27.248301 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 16:40:27.263093 11822 raft_consensus.cc:1261] T f2149cdb30e24ce593a6f073a4e61edb P c31f41646fab4fbe86444d408879c9d0 [term 1 FOLLOWER]: Refusing update from remote peer 1b05ed463c35461c9067db7784ca88a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:40:27.263958 12166 consensus_queue.cc:1035] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31f41646fab4fbe86444d408879c9d0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:40:27.272421 11943 raft_consensus.cc:1261] T f2149cdb30e24ce593a6f073a4e61edb P 7395ddd784a545f191d8957cbd45be31 [term 1 FOLLOWER]: Refusing update from remote peer 1b05ed463c35461c9067db7784ca88a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:40:27.273118 12166 consensus_queue.cc:1035] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7395ddd784a545f191d8957cbd45be31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 27 16:40:27 dist-test-slave-z40b krb5kdc[11178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:40:27.285949 12169 mvcc.cc:205] Tried to move back new op lower bound from 7333999522860449792 to 7333999522733662208. Current Snapshot: MvccSnapshot[applied={T|T < 7333999522860449792}]
I20260927 16:40:27.286962 12171 mvcc.cc:205] Tried to move back new op lower bound from 7333999522860449792 to 7333999522733662208. Current Snapshot: MvccSnapshot[applied={T|T < 7333999522860449792}]
I20260927 16:40:27.295862 12173 mvcc.cc:205] Tried to move back new op lower bound from 7333999522860449792 to 7333999522733662208. Current Snapshot: MvccSnapshot[applied={T|T < 7333999522860449792}]
I20260927 16:40:27.305132 11653 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:40:27.305289 11653 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:40:27.308072 11653 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 93c3aeb1d8724a8eb65523d059bb50ad: Sending DeleteTablet for 3 replicas of tablet f2149cdb30e24ce593a6f073a4e61edb
I20260927 16:40:27.309041 11938 tablet_service.cc:1514] Processing DeleteTablet for tablet f2149cdb30e24ce593a6f073a4e61edb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:40:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49350
I20260927 16:40:27.309499 11789 tablet_service.cc:1514] Processing DeleteTablet for tablet f2149cdb30e24ce593a6f073a4e61edb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:40:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36280
I20260927 16:40:27.310465 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 11735
I20260927 16:40:27.313305 12070 tablet_service.cc:1514] Processing DeleteTablet for tablet f2149cdb30e24ce593a6f073a4e61edb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:40:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45842
I20260927 16:40:27.323772 12179 tablet_replica.cc:330] T f2149cdb30e24ce593a6f073a4e61edb P 7395ddd784a545f191d8957cbd45be31: stopping tablet replica
I20260927 16:40:27.324146 12179 raft_consensus.cc:2229] T f2149cdb30e24ce593a6f073a4e61edb P 7395ddd784a545f191d8957cbd45be31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:40:27.324554 12179 raft_consensus.cc:2258] T f2149cdb30e24ce593a6f073a4e61edb P 7395ddd784a545f191d8957cbd45be31 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 16:40:27.324988 11641 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40403 (error 108)
I20260927 16:40:27.326781 12179 ts_tablet_manager.cc:1905] T f2149cdb30e24ce593a6f073a4e61edb P 7395ddd784a545f191d8957cbd45be31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:40:27.328925 12181 tablet_replica.cc:330] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9: stopping tablet replica
I20260927 16:40:27.329276 12181 raft_consensus.cc:2229] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:40:27.329797 12179 ts_tablet_manager.cc:1918] T f2149cdb30e24ce593a6f073a4e61edb P 7395ddd784a545f191d8957cbd45be31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:40:27.330412 12181 raft_consensus.cc:2258] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:40:27.331334 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 11874
I20260927 16:40:27.336100 12181 ts_tablet_manager.cc:1905] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:40:27.351622 12153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:40:27.351193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40403 (local address 127.0.0.1:36288)
0927 16:40:27.351435 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 16:40:27.351523 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260927 16:40:27.351948 11641 catalog_manager.cc:4584] TS c31f41646fab4fbe86444d408879c9d0 (127.0.98.1:40403): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f2149cdb30e24ce593a6f073a4e61edb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40403: connect: Connection refused (error 111)
I20260927 16:40:27.353143 12181 ts_tablet_manager.cc:1918] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:40:27.354827 12181 log.cc:1196] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790527121515808-392-0/minicluster-data/ts-2/wal/wals/f2149cdb30e24ce593a6f073a4e61edb
I20260927 16:40:27.355472 12181 ts_tablet_manager.cc:1939] T f2149cdb30e24ce593a6f073a4e61edb P 1b05ed463c35461c9067db7784ca88a9: Deleting consensus metadata
W20260927 16:40:27.357758 12153 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:40:27.357117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41555 (local address 127.0.0.1:49352)
0927 16:40:27.357269 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 16:40:27.357284 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:40:27.357367 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:40:27.357636 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41555: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 16:40:27.359155 11641 catalog_manager.cc:4841] TS 1b05ed463c35461c9067db7784ca88a9 (127.0.98.3:39189): tablet f2149cdb30e24ce593a6f073a4e61edb (table test-table [id=7a2d940fccf04d6da1e2e2ebf30d77bc]) successfully deleted
I20260927 16:40:27.359573 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 12009
I20260927 16:40:27.366340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 11622
I20260927 16:40:27.376911 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 16:40:27.720542 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 16:40:27.720654 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 16:40:28.065650 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 16:40:28.065 UTC [11208] LOG: received smart shutdown request
2026-09-27 16:40:28.069 UTC [11208] LOG: background worker "logical replication launcher" (PID 11217) exited with exit code 1
2026-09-27 16:40:28.069 UTC [11212] LOG: shutting down
2026-09-27 16:40:28.218 UTC [11208] LOG: database system is shut down
2026-09-27T16:40:28Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44404 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:40:28 dist-test-slave-z40b krb5kdc[12192](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:40:28 dist-test-slave-z40b krb5kdc[12192](info): set up 2 sockets
Sep 27 16:40:28 dist-test-slave-z40b krb5kdc[12192](info): commencing operation
krb5kdc: starting...
W20260927 16:40:29.322885 392 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:40:29 dist-test-slave-z40b krb5kdc[12192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:40:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:40:29Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 16:40:29.473112 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskmlIKrs/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/postgres -l logfile start
2026-09-27 16:40:31.727 UTC [12222] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 16:40:31.727 UTC [12222] LOG: listening on IPv4 address "127.0.98.20", port 34715
2026-09-27 16:40:31.731 UTC [12222] LOG: listening on Unix socket "/tmp/.s.PGSQL.34715"
2026-09-27 16:40:31.750 UTC [12225] LOG: database system was shut down at 2026-09-27 16:40:30 UTC
2026-09-27 16:40:31.753 UTC [12222] LOG: database system is ready to accept connections
I20260927 16:40:32.662994 392 mini_postgres.cc:96] Postgres bound to 34715
2026-09-27 16:40:32.669 UTC [12233] FATAL: database "slave" does not exist
127.0.98.20:34715 - accepting connections
I20260927 16:40:32.669548 392 mini_ranger.cc:161] Starting Ranger...
I20260927 16:40:32.683545 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 16:40:33.115900 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 16:40:33.115984 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ranger-admin
2026-09-27 16:40:33,369 [I] DB FLAVOR :POSTGRES
2026-09-27 16:40:33,369 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 16:40:33,369 [I] Checking connection..
2026-09-27 16:40:33,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 16:40:33,614 [I] Checking connection passed.
2026-09-27 16:40:33,614 [I] --------- Verifying version history table ---------
2026-09-27 16:40:33,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 16:40:33,862 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 16:40:33,862 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 16:40:34,085 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 16:40:34,085 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 16:40:34,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34715/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 16:40:34.327 UTC [12297] WARNING: there is no transaction in progress
2026-09-27 16:40:34,333 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 16:40:34,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 16:40:34,561 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 16:40:34,561 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 16:40:34,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 16:40:34,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 16:40:35,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-z40b.c.gcp-upstream.internal','N') ;"
2026-09-27 16:40:35,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 16:40:35,536 [I] Table x_portal_user does not exist in database ranger
2026-09-27 16:40:35,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 16:40:35,759 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 16:40:35,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 16:40:35,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 16:40:36,203 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 16:40:36,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34715/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 16:40:37.064 UTC [12405] WARNING: there is no transaction in progress
2026-09-27 16:40:37.078 UTC [12405] WARNING: there is no transaction in progress
2026-09-27 16:40:37.093 UTC [12405] WARNING: there is no transaction in progress
2026-09-27 16:40:37.108 UTC [12405] WARNING: there is no transaction in progress
2026-09-27 16:40:37.126 UTC [12405] WARNING: there is no transaction in progress
2026-09-27 16:40:37.144 UTC [12405] WARNING: there is no transaction in progress
2026-09-27 16:40:37.204 UTC [12405] WARNING: there is no transaction in progress
2026-09-27 16:40:37.211 UTC [12405] WARNING: there is no transaction in progress
2026-09-27 16:40:37.224 UTC [12405] WARNING: there is no transaction in progress
2026-09-27 16:40:37.233 UTC [12405] WARNING: there is no transaction in progress
2026-09-27 16:40:37.243 UTC [12405] WARNING: there is no transaction in progress
2026-09-27 16:40:37.255 UTC [12405] WARNING: there is no transaction in progress
2026-09-27 16:40:37.264 UTC [12405] WARNING: there is no transaction in progress
2026-09-27 16:40:37.271 UTC [12405] WARNING: there is no transaction in progress
2026-09-27 16:40:37.277 UTC [12405] WARNING: there is no transaction in progress
2026-09-27 16:40:37.283 UTC [12405] WARNING: there is no transaction in progress
2026-09-27 16:40:37,870 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 16:40:37,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 16:40:38,107 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 16:40:38,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 16:40:38,334 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 16:40:38,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 16:40:38,584 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 16:40:38,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 16:40:38,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 16:40:39,040 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-z40b.c.gcp-upstream.internal';"
2026-09-27 16:40:39,252 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 16:40:39,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 16:40:39,480 [I] DB_PATCHES have already been applied
I20260927 16:40:39.485327 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskmlIKrs/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 16:40:39.485378 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 16:40:39.488649 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:48893
Sep 27, 2026 4:40:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 4:40:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 4:40:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 4:40:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 4:40:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 4:40:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 4:40:40 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 16:40:40.565169 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.896s user 0.004s sys 0.005s
Sep 27, 2026 4:40:59 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 4:40:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 4:40:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 4:41:00 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 4:41:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 4:41:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 4:41:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 4:41:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 4:41:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 4:41:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 4:41:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 4:41:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 4:41:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 4:41:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 4:41:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 4:41:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 4:41:00 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 4:41:00 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 16:41:01.440462 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 16:41:01 dist-test-slave-z40b krb5kdc[12192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527261, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 16:41:01.516275 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 16:41:01.525496 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 16:41:02.337374 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 16:41:02.337455 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ranger-kms
I20260927 16:41:02.337478 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:34715
2026-09-27 16:41:02,502 [I] DB FLAVOR :POSTGRES
2026-09-27 16:41:02,503 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 16:41:02,503 [I] Checking connection
2026-09-27 16:41:02,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34715/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 16:41:02,727 [I] connection success
2026-09-27 16:41:02,727 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 16:41:02,727 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 16:41:02,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34715/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 16:41:02,968 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 16:41:02,968 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 16:41:02,968 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 16:41:02,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:34715/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 16:41:03,200 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 16:41:03.636848 392 mini_ranger_kms.cc:310] Created kms service
I20260927 16:41:03.836858 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 16:41:03.878755 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 16:41:03 dist-test-slave-z40b krb5kdc[12192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527261, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:41:04.239673 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 16:41:04.239811 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskmlIKrs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskmlIKrs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 16:41:04.239832 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 16:41:04.243177 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12585
I20260927 16:41:04.243275 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:55175
Sep 27, 2026 4:41:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 4:41:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 4:41:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 4:41:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 4:41:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 4:41:05 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 16:41:06 dist-test-slave-z40b krb5kdc[12192](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790527266, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:06.366976 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.851s user 0.017s sys 0.132s
I20260927 16:41:06.367134 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 16:41:06.367237 392 mini_ranger_kms.cc:401] 127.0.98.20:55175/kms/v1/keys
Sep 27 16:41:06 dist-test-slave-z40b krb5kdc[12192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527266, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 16:41:07 dist-test-slave-z40b krb5kdc[12192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527266, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 16:41:09 dist-test-slave-z40b krb5kdc[12192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:41:09.200441 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37825
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37825
--ranger_config_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55175/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:41:09.307020 12636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:09.307345 12636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:09.307416 12636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:09.311321 12636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:41:09.311367 12636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:09.311385 12636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:09.311400 12636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:41:09.311414 12636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:41:09.316660 12636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:55175/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37825
--ranger_config_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:09.317591 12636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:09.318576 12636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:41:09.326812 12641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:09.326831 12642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:09.326825 12644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:09.327167 12636 server_base.cc:1034] running on GCE node
I20260927 16:41:09.327730 12636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:09.328740 12636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:09.329916 12636 hybrid_clock.cc:648] HybridClock initialized: now 1790527269329901 us; error 31 us; skew 500 ppm
Sep 27 16:41:09 dist-test-slave-z40b krb5kdc[12192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:09.332705 12636 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:41:09.333580 12636 webserver.cc:460] Webserver started at http://127.0.98.62:37719/ using document root <none> and password file <none>
I20260927 16:41:09.334060 12636 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:09.334136 12636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:41:09.334322 12636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 16:41:09 dist-test-slave-z40b krb5kdc[12192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:41:09.392519 12636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "741cb023496f4e66935d9d82d6e61f3a"
format_stamp: "Formatted at 2026-09-27 16:41:09 on dist-test-slave-z40b"
server_key: "d7f809f7595775a2eefbcc52a9892437"
server_key_iv: "063d23158f3f0d19de6d9230445cb170"
server_key_version: "kuduclusterkey@0"
I20260927 16:41:09.393136 12636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "741cb023496f4e66935d9d82d6e61f3a"
format_stamp: "Formatted at 2026-09-27 16:41:09 on dist-test-slave-z40b"
server_key: "d7f809f7595775a2eefbcc52a9892437"
server_key_iv: "063d23158f3f0d19de6d9230445cb170"
server_key_version: "kuduclusterkey@0"
I20260927 16:41:09.397495 12636 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 16:41:09.414140 12651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:09.415711 12636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:41:09.415830 12636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "741cb023496f4e66935d9d82d6e61f3a"
format_stamp: "Formatted at 2026-09-27 16:41:09 on dist-test-slave-z40b"
server_key: "d7f809f7595775a2eefbcc52a9892437"
server_key_iv: "063d23158f3f0d19de6d9230445cb170"
server_key_version: "kuduclusterkey@0"
I20260927 16:41:09.415966 12636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:09.446527 12636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:09.449692 12636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:09.449893 12636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:09.457937 12636 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37825
I20260927 16:41:09.457955 12703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37825 every 8 connection(s)
I20260927 16:41:09.459003 12636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:41:09.461711 12704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:41:09.466861 12704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a: Bootstrap starting.
I20260927 16:41:09.467011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 12636
I20260927 16:41:09.467183 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:41:09.467480 392 external_mini_cluster.cc:1428] Setting key fdd223dd737d5f88c4d1e67883a30e1d
I20260927 16:41:09.469622 12704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a: Neither blocks nor log segments found. Creating new log.
I20260927 16:41:09.470480 12704 log.cc:824] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a: Log is configured to *not* fsync() on all Append() calls
I20260927 16:41:09.472374 12704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a: No bootstrap required, opened a new log
I20260927 16:41:09.476328 12704 raft_consensus.cc:348] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741cb023496f4e66935d9d82d6e61f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37825 } }
I20260927 16:41:09.476554 12704 raft_consensus.cc:374] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:41:09.476631 12704 raft_consensus.cc:729] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 741cb023496f4e66935d9d82d6e61f3a, State: Initialized, Role: FOLLOWER
I20260927 16:41:09.477099 12704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741cb023496f4e66935d9d82d6e61f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37825 } }
Sep 27 16:41:09 dist-test-slave-z40b krb5kdc[12192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:41:09.477244 12704 raft_consensus.cc:388] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:41:09.477298 12704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:41:09.477421 12704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:41:09.478508 12704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741cb023496f4e66935d9d82d6e61f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37825 } }
I20260927 16:41:09.478855 12704 leader_election.cc:302] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 741cb023496f4e66935d9d82d6e61f3a; no voters:
I20260927 16:41:09.479243 12704 leader_election.cc:288] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:41:09.479336 12709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:41:09.479591 12709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [term 1 LEADER]: Becoming Leader. State: Replica: 741cb023496f4e66935d9d82d6e61f3a, State: Running, Role: LEADER
I20260927 16:41:09.480552 12704 sys_catalog.cc:564] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:41:09.479954 12709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741cb023496f4e66935d9d82d6e61f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37825 } }
I20260927 16:41:09.483251 12707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:09.468818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52070 (local address 127.0.98.62:37825)
0927 16:41:09.469201 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:09.469217 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:09.469253 (+ 36us) server_negotiation.cc:407] Connection header received
0927 16:41:09.470098 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:09.470119 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:09.470516 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:09.470886 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:09.471700 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:09.472961 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:09.474206 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:09.474871 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:09.475020 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:09.477843 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:09.477849 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:09.477864 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:09.477893 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:09.479952 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:09.480426 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:09.480432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:09.480438 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:09.480558 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:09.480871 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:09.480874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:09.480877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:09.481410 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:09.481627 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:09.482136 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 16:41:09.482380 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":119,"threads_started":1}
I20260927 16:41:09.483369 12711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 741cb023496f4e66935d9d82d6e61f3a. Latest consensus state: current_term: 1 leader_uuid: "741cb023496f4e66935d9d82d6e61f3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741cb023496f4e66935d9d82d6e61f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37825 } } }
I20260927 16:41:09.483479 12711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [sys.catalog]: This master's current role is: LEADER
I20260927 16:41:09.483752 12710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "741cb023496f4e66935d9d82d6e61f3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741cb023496f4e66935d9d82d6e61f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37825 } } }
I20260927 16:41:09.483824 12710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [sys.catalog]: This master's current role is: LEADER
I20260927 16:41:09.483385 12704 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 16:41:09.488286 12723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:41:09.492137 12723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:41:09.504190 12723 catalog_manager.cc:1347] Generated new cluster ID: d3f436b2812e411bb77bc296c7b71573
I20260927 16:41:09.504242 12723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:41:09.521904 12723 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:41:09.523258 12723 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:41:09.537421 12723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a: Generated new TSK 0
I20260927 16:41:09.538329 12723 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:41:10.165763 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55175/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37825
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:41:10.278995 12749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:10.279271 12749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:10.279341 12749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:10.283614 12749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:10.283692 12749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:10.283806 12749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:41:10.289693 12749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:55175/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:10.291095 12749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:10.292095 12749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:41:10.301489 12754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:10.301491 12755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:10.301666 12749 server_base.cc:1034] running on GCE node
W20260927 16:41:10.301505 12757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:10.302083 12749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:10.302727 12749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:10.303927 12749 hybrid_clock.cc:648] HybridClock initialized: now 1790527270303900 us; error 42 us; skew 500 ppm
Sep 27 16:41:10 dist-test-slave-z40b krb5kdc[12192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:10.307358 12749 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:41:10.308115 12749 webserver.cc:460] Webserver started at http://127.0.98.1:42521/ using document root <none> and password file <none>
I20260927 16:41:10.308669 12749 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:10.308750 12749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:41:10.308950 12749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 16:41:10 dist-test-slave-z40b krb5kdc[12192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:41:10.325929 12749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/data/instance:
uuid: "362b074721ca4dd7b81a62b40b601c3f"
format_stamp: "Formatted at 2026-09-27 16:41:10 on dist-test-slave-z40b"
server_key: "7782cd145021fe7fa34feb49bd108820"
server_key_iv: "71d4b186af73fcdd11c6f49953ddd25b"
server_key_version: "kuduclusterkey@0"
I20260927 16:41:10.326624 12749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/wal/instance:
uuid: "362b074721ca4dd7b81a62b40b601c3f"
format_stamp: "Formatted at 2026-09-27 16:41:10 on dist-test-slave-z40b"
server_key: "7782cd145021fe7fa34feb49bd108820"
server_key_iv: "71d4b186af73fcdd11c6f49953ddd25b"
server_key_version: "kuduclusterkey@0"
I20260927 16:41:10.331665 12749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 16:41:10.353302 12766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:10.355296 12749 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 16:41:10.355465 12749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "362b074721ca4dd7b81a62b40b601c3f"
format_stamp: "Formatted at 2026-09-27 16:41:10 on dist-test-slave-z40b"
server_key: "7782cd145021fe7fa34feb49bd108820"
server_key_iv: "71d4b186af73fcdd11c6f49953ddd25b"
server_key_version: "kuduclusterkey@0"
I20260927 16:41:10.355681 12749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:10.381644 12749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:10.384981 12749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:10.385171 12749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:10.385782 12749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:41:10.386636 12749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:41:10.386703 12749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:10.386783 12749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:41:10.386821 12749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:10.398010 12749 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41715
I20260927 16:41:10.399101 12749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:41:10.403705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 12749
I20260927 16:41:10.403812 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:41:10.404103 392 external_mini_cluster.cc:1428] Setting key 5da8e73e7a0bd4558965c163973aa20a
I20260927 16:41:10.403379 12879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41715 every 8 connection(s)
Sep 27 16:41:10 dist-test-slave-z40b krb5kdc[12192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 16:41:10.427068 12884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:10.410157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49197 (local address 127.0.98.62:37825)
0927 16:41:10.412974 (+ 2817us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:10.412982 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:10.412995 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:41:10.413106 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:10.413110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:10.413157 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:10.413239 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:10.414635 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:10.415553 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:10.417325 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:10.417613 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:10.417693 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:10.422581 (+ 4888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:10.422585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:10.422588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:10.422611 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:10.424668 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:10.425282 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:10.425289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:10.425292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:10.425389 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:10.425805 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:10.425809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:10.425811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:10.426018 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:10.426085 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:10.426748 (+ 663us) server_negotiation.cc:299] Negotiation successful
0927 16:41:10.426869 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2743,"thread_start_us":97,"threads_started":1}
I20260927 16:41:10.428956 12883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:10.409155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37825 (local address 127.0.98.1:49197)
0927 16:41:10.411513 (+ 2358us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:10.411576 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:10.412586 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:10.413512 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:10.413520 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:10.413855 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:41:10.414464 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:10.414480 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:10.415709 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:10.415714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:10.417189 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:10.417202 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:10.417767 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:10.417773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:10.418008 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:10.420535 (+ 2527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:41:10.420558 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:41:10.422395 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:41:10.424797 (+ 2402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:10.424803 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:10.424817 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:10.425138 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:10.425497 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:10.425500 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:10.425502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:10.425689 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:10.426163 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:41:10.426169 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:41:10.426445 (+ 276us) client_negotiation.cc:769] Sending connection context
0927 16:41:10.427639 (+ 1194us) client_negotiation.cc:240] Negotiation successful
0927 16:41:10.427975 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2147,"thread_start_us":135,"threads_started":1}
I20260927 16:41:10.430364 12880 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37825
I20260927 16:41:10.430614 12880 heartbeater.cc:464] Registering TS with master...
I20260927 16:41:10.431157 12880 heartbeater.cc:511] Master 127.0.98.62:37825 requested a full tablet report, sending...
I20260927 16:41:10.432807 12668 ts_manager.cc:194] Registered new tserver with Master: 362b074721ca4dd7b81a62b40b601c3f (127.0.98.1:41715)
I20260927 16:41:10.434252 12668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:41:10.446591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55175/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37825
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 16:41:10 dist-test-slave-z40b krb5kdc[12192](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790527270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 16:41:10.597131 12888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:10.597455 12888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:10.597513 12888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:10.603116 12888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:10.603186 12888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:10.603293 12888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:41:10.610425 12888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:55175/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:10.611670 12888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:10.612851 12888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:41:10.624523 12895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:10.624634 12897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:10.624514 12894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:10.624842 12888 server_base.cc:1034] running on GCE node
I20260927 16:41:10.625293 12888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:10.625903 12888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:10.627103 12888 hybrid_clock.cc:648] HybridClock initialized: now 1790527270627073 us; error 49 us; skew 500 ppm
Sep 27 16:41:10 dist-test-slave-z40b krb5kdc[12192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:10.629733 12888 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:41:10.630620 12888 webserver.cc:460] Webserver started at http://127.0.98.2:33209/ using document root <none> and password file <none>
I20260927 16:41:10.631151 12888 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:10.631282 12888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:41:10.631567 12888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 16:41:10 dist-test-slave-z40b krb5kdc[12192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:41:10.646833 12888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/data/instance:
uuid: "48eb0c04a8d7477f9f482420f3dd71a6"
format_stamp: "Formatted at 2026-09-27 16:41:10 on dist-test-slave-z40b"
server_key: "315d86b4d8f69e47539b141b3e6836d9"
server_key_iv: "1ab9ad4850e7e950b911c3f7e8c14dff"
server_key_version: "kuduclusterkey@0"
I20260927 16:41:10.647585 12888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/wal/instance:
uuid: "48eb0c04a8d7477f9f482420f3dd71a6"
format_stamp: "Formatted at 2026-09-27 16:41:10 on dist-test-slave-z40b"
server_key: "315d86b4d8f69e47539b141b3e6836d9"
server_key_iv: "1ab9ad4850e7e950b911c3f7e8c14dff"
server_key_version: "kuduclusterkey@0"
I20260927 16:41:10.660142 12888 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.003s sys 0.001s
I20260927 16:41:10.685081 12904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:10.686621 12888 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260927 16:41:10.686808 12888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "48eb0c04a8d7477f9f482420f3dd71a6"
format_stamp: "Formatted at 2026-09-27 16:41:10 on dist-test-slave-z40b"
server_key: "315d86b4d8f69e47539b141b3e6836d9"
server_key_iv: "1ab9ad4850e7e950b911c3f7e8c14dff"
server_key_version: "kuduclusterkey@0"
I20260927 16:41:10.687001 12888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:10.722504 12888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:10.725991 12888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:10.726212 12888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:10.726961 12888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:41:10.728049 12888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:41:10.728161 12888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:10.728272 12888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:41:10.728336 12888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:10.737740 12888 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44653
I20260927 16:41:10.738742 12888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:41:10.740752 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 12888
I20260927 16:41:10.740880 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:41:10.741254 392 external_mini_cluster.cc:1428] Setting key 1b77ac9ef2dcb46d79b13e3114421cf3
I20260927 16:41:10.750738 13017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44653 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 27 16:41:10 dist-test-slave-z40b krb5kdc[12192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 16:41:10.762247 12884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:10.749194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49403 (local address 127.0.98.62:37825)
0927 16:41:10.749318 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:10.749323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:10.750452 (+ 1129us) server_negotiation.cc:407] Connection header received
0927 16:41:10.751501 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:10.751504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:10.751557 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:10.751674 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:10.752902 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:10.753539 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:10.754709 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:10.755061 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:10.755156 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:10.758044 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:10.758049 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:10.758051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:10.758078 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:10.759831 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:10.760493 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:10.760497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:10.760501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:10.760587 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:10.761018 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:10.761022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:10.761024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:10.761254 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:10.761328 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:10.761984 (+ 656us) server_negotiation.cc:299] Negotiation successful
0927 16:41:10.762124 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:41:10.763177 13021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:10.749699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37825 (local address 127.0.98.2:49403)
0927 16:41:10.750301 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:10.750360 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:10.751308 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:10.751931 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:10.751940 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:10.752235 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:41:10.752710 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:10.752746 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:10.753670 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:10.753673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:10.754583 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:10.754592 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:10.755342 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:10.755347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:10.755567 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:10.756227 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:41:10.756245 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:41:10.757851 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:41:10.759985 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:10.760021 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:10.760036 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:10.760369 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:10.760722 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:10.760725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:10.760727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:10.760898 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:10.761411 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:41:10.761417 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:41:10.761725 (+ 308us) client_negotiation.cc:769] Sending connection context
0927 16:41:10.761928 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 16:41:10.762227 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":159,"threads_started":1}
I20260927 16:41:10.764266 13018 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37825
I20260927 16:41:10.764521 13018 heartbeater.cc:464] Registering TS with master...
I20260927 16:41:10.764976 13018 heartbeater.cc:511] Master 127.0.98.62:37825 requested a full tablet report, sending...
I20260927 16:41:10.765947 12668 ts_manager.cc:194] Registered new tserver with Master: 48eb0c04a8d7477f9f482420f3dd71a6 (127.0.98.2:44653)
I20260927 16:41:10.766562 12668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:41:10.778303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55175/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37825
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:41:10.908329 13023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:10.908568 13023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:10.908631 13023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:10.912700 13023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:10.912797 13023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:10.912911 13023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:41:10.917959 13023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:55175/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:10.918883 13023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:10.919818 13023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:41:10.931493 13031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:10.931552 13028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:10.931741 13029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:10.932088 13023 server_base.cc:1034] running on GCE node
I20260927 16:41:10.932468 13023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:10.933060 13023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:10.934234 13023 hybrid_clock.cc:648] HybridClock initialized: now 1790527270934183 us; error 64 us; skew 500 ppm
Sep 27 16:41:10 dist-test-slave-z40b krb5kdc[12192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:10.937980 13023 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:41:10.938941 13023 webserver.cc:460] Webserver started at http://127.0.98.3:45701/ using document root <none> and password file <none>
I20260927 16:41:10.939659 13023 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:10.939795 13023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:41:10.940058 13023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 16:41:10 dist-test-slave-z40b krb5kdc[12192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:41:10.957454 13023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/data/instance:
uuid: "7ad61091e523447d862a1f4447258ba8"
format_stamp: "Formatted at 2026-09-27 16:41:10 on dist-test-slave-z40b"
server_key: "00a912b48e476a28c5a441c229e7a7da"
server_key_iv: "9b37e53dfbe18e1c82412270a178e789"
server_key_version: "kuduclusterkey@0"
I20260927 16:41:10.958328 13023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7ad61091e523447d862a1f4447258ba8"
format_stamp: "Formatted at 2026-09-27 16:41:10 on dist-test-slave-z40b"
server_key: "00a912b48e476a28c5a441c229e7a7da"
server_key_iv: "9b37e53dfbe18e1c82412270a178e789"
server_key_version: "kuduclusterkey@0"
Sep 27 16:41:10 dist-test-slave-z40b krb5kdc[12192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:41:10.963579 13023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 16:41:10.980166 13038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:10.982316 13023 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260927 16:41:10.982451 13023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "7ad61091e523447d862a1f4447258ba8"
format_stamp: "Formatted at 2026-09-27 16:41:10 on dist-test-slave-z40b"
server_key: "00a912b48e476a28c5a441c229e7a7da"
server_key_iv: "9b37e53dfbe18e1c82412270a178e789"
server_key_version: "kuduclusterkey@0"
I20260927 16:41:10.982669 13023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:11.011559 13023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:11.014972 13023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:11.015192 13023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:11.015930 13023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:41:11.016957 13023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:41:11.017057 13023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:11.017148 13023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:41:11.017203 13023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:11.027230 13023 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39557
I20260927 16:41:11.028448 13023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:41:11.035457 13151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39557 every 8 connection(s)
I20260927 16:41:11.035754 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 13023
I20260927 16:41:11.035861 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:41:11.036137 392 external_mini_cluster.cc:1428] Setting key 2a83389ea46d4002ef8e6be803cd8df0
Sep 27 16:41:11 dist-test-slave-z40b krb5kdc[12192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:41:11.052107 12884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:11.031392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56223 (local address 127.0.98.62:37825)
0927 16:41:11.031509 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:11.031513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:11.039870 (+ 8357us) server_negotiation.cc:407] Connection header received
0927 16:41:11.040989 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:11.040992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:11.041041 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:11.041154 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:11.042388 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:11.043185 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:11.044749 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:11.045106 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:11.045197 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:11.047969 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:11.047973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:11.047976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:11.048002 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:11.049809 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:11.050400 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:11.050404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:11.050406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:11.050496 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:11.050879 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:11.050883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:11.050884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:11.051099 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:11.051208 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:11.051828 (+ 620us) server_negotiation.cc:299] Negotiation successful
0927 16:41:11.051945 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:41:11.053248 13154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:11.031077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37825 (local address 127.0.98.3:56223)
0927 16:41:11.039712 (+ 8635us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:11.039762 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:11.040767 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:11.041330 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:11.041338 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:11.041650 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:41:11.042227 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:11.042245 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:11.043311 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:11.043315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:11.044644 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:11.044654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:11.045260 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:11.045265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:11.045442 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:11.046114 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:41:11.046133 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:41:11.047795 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:41:11.049926 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:11.049931 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:11.049962 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:11.050293 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:11.050603 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:11.050606 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:11.050608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:11.050777 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:11.051244 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:41:11.051250 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:41:11.051555 (+ 305us) client_negotiation.cc:769] Sending connection context
0927 16:41:11.051772 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 16:41:11.052057 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8398,"thread_start_us":447,"threads_started":1}
I20260927 16:41:11.054330 13152 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37825
I20260927 16:41:11.054946 13152 heartbeater.cc:464] Registering TS with master...
I20260927 16:41:11.055500 13152 heartbeater.cc:511] Master 127.0.98.62:37825 requested a full tablet report, sending...
I20260927 16:41:11.056576 12668 ts_manager.cc:194] Registered new tserver with Master: 7ad61091e523447d862a1f4447258ba8 (127.0.98.3:39557)
I20260927 16:41:11.057235 12668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56223
I20260927 16:41:11.068363 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:41:11.069089 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 12749
I20260927 16:41:11.074378 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 12888
I20260927 16:41:11.079574 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 13023
I20260927 16:41:11.084669 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 12636
I20260927 16:41:11.092844 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37825
--webserver_interface=127.0.98.62
--webserver_port=37719
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37825
--ranger_config_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55175/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55175/kms with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:41:11.195358 13155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:11.195583 13155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:11.195628 13155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:11.199424 13155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:41:11.199467 13155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:11.199486 13155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:11.199501 13155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:41:11.199515 13155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:41:11.204602 13155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55175/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37825
--ranger_config_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:11.205469 13155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:11.206348 13155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:41:11.214425 13160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:11.214460 13161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:11.214432 13163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:11.214624 13155 server_base.cc:1034] running on GCE node
I20260927 16:41:11.215306 13155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:11.216264 13155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:11.217424 13155 hybrid_clock.cc:648] HybridClock initialized: now 1790527271217394 us; error 45 us; skew 500 ppm
Sep 27 16:41:11 dist-test-slave-z40b krb5kdc[12192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:11.220114 13155 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:41:11.220916 13155 webserver.cc:460] Webserver started at http://127.0.98.62:37719/ using document root <none> and password file <none>
I20260927 16:41:11.221385 13155 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:11.221446 13155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 16:41:11 dist-test-slave-z40b krb5kdc[12192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:41:11.280860 13155 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:41:11.283097 13171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:11.284499 13155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 16:41:11.284598 13155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "741cb023496f4e66935d9d82d6e61f3a"
format_stamp: "Formatted at 2026-09-27 16:41:09 on dist-test-slave-z40b"
server_key: "d7f809f7595775a2eefbcc52a9892437"
server_key_iv: "063d23158f3f0d19de6d9230445cb170"
server_key_version: "kuduclusterkey@0"
I20260927 16:41:11.285166 13155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:11.316445 13155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:11.319378 13155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:11.319567 13155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:11.326252 13155 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37825
I20260927 16:41:11.326259 13223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37825 every 8 connection(s)
I20260927 16:41:11.327394 13155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:41:11.329293 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 13155
I20260927 16:41:11.329698 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55175/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41715
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37825
--webserver_port=42521
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55175/kms with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:41:11.331643 13224 sys_catalog.cc:263] Verifying existing consensus state
I20260927 16:41:11.333277 13224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a: Bootstrap starting.
I20260927 16:41:11.348621 13224 log.cc:824] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a: Log is configured to *not* fsync() on all Append() calls
I20260927 16:41:11.354758 13224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:41:11.355171 13224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a: Bootstrap complete.
I20260927 16:41:11.359122 13224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741cb023496f4e66935d9d82d6e61f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37825 } }
I20260927 16:41:11.359578 13224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 741cb023496f4e66935d9d82d6e61f3a, State: Initialized, Role: FOLLOWER
I20260927 16:41:11.360051 13224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741cb023496f4e66935d9d82d6e61f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37825 } }
I20260927 16:41:11.360190 13224 raft_consensus.cc:388] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:41:11.360275 13224 raft_consensus.cc:482] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:41:11.360385 13224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [term 1 FOLLOWER]: Advancing to term 2
I20260927 16:41:11.362192 13224 raft_consensus.cc:504] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741cb023496f4e66935d9d82d6e61f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37825 } }
I20260927 16:41:11.362506 13224 leader_election.cc:302] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 741cb023496f4e66935d9d82d6e61f3a; no voters:
I20260927 16:41:11.362797 13224 leader_election.cc:288] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 16:41:11.362923 13228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [term 2 FOLLOWER]: Leader election won for term 2
I20260927 16:41:11.363168 13228 raft_consensus.cc:686] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [term 2 LEADER]: Becoming Leader. State: Replica: 741cb023496f4e66935d9d82d6e61f3a, State: Running, Role: LEADER
I20260927 16:41:11.363547 13228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741cb023496f4e66935d9d82d6e61f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37825 } }
I20260927 16:41:11.364007 13224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:41:11.365053 13230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 741cb023496f4e66935d9d82d6e61f3a. Latest consensus state: current_term: 2 leader_uuid: "741cb023496f4e66935d9d82d6e61f3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741cb023496f4e66935d9d82d6e61f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37825 } } }
I20260927 16:41:11.365212 13230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [sys.catalog]: This master's current role is: LEADER
I20260927 16:41:11.365437 13229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "741cb023496f4e66935d9d82d6e61f3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741cb023496f4e66935d9d82d6e61f3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37825 } } }
I20260927 16:41:11.365509 13229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a [sys.catalog]: This master's current role is: LEADER
I20260927 16:41:11.365537 13237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:41:11.368589 13237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:41:11.369045 13224 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 16:41:11.369284 13237 catalog_manager.cc:1259] Loaded cluster ID: d3f436b2812e411bb77bc296c7b71573
I20260927 16:41:11.369382 13237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:41:11.371752 13237 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:41:11.372617 13237 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a: Loaded TSK: 0
I20260927 16:41:11.373883 13237 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 16:41:11.463413 13226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:11.463809 13226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:11.463903 13226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:11.470336 13226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:11.470400 13226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:11.470485 13226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:41:11.478053 13226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55175/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42521
--tserver_master_addrs=127.0.98.62:37825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:11.479324 13226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:11.480543 13226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:41:11.492256 13264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:11.492266 13261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:11.492274 13262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:11.493413 13226 server_base.cc:1034] running on GCE node
I20260927 16:41:11.493860 13226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:11.494590 13226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:11.495779 13226 hybrid_clock.cc:648] HybridClock initialized: now 1790527271495758 us; error 34 us; skew 500 ppm
Sep 27 16:41:11 dist-test-slave-z40b krb5kdc[12192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:11.499163 13226 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:41:11.500248 13226 webserver.cc:460] Webserver started at http://127.0.98.1:42521/ using document root <none> and password file <none>
I20260927 16:41:11.501365 13226 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:11.501473 13226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 16:41:11 dist-test-slave-z40b krb5kdc[12192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:41:11.524776 13226 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.002s sys 0.000s
I20260927 16:41:11.527814 13272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:11.529332 13226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:41:11.529453 13226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "362b074721ca4dd7b81a62b40b601c3f"
format_stamp: "Formatted at 2026-09-27 16:41:10 on dist-test-slave-z40b"
server_key: "7782cd145021fe7fa34feb49bd108820"
server_key_iv: "71d4b186af73fcdd11c6f49953ddd25b"
server_key_version: "kuduclusterkey@0"
I20260927 16:41:11.530169 13226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:11.566202 13226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:11.569690 13226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:11.569873 13226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:11.570609 13226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:41:11.571667 13226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:41:11.571765 13226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:11.571898 13226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:41:11.571985 13226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:11.582736 13226 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41715
I20260927 16:41:11.582789 13385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41715 every 8 connection(s)
I20260927 16:41:11.584059 13226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:41:11.586912 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 13226
I20260927 16:41:11.587453 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55175/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44653
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37825
--webserver_port=33209
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55175/kms with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 16:41:11 dist-test-slave-z40b krb5kdc[12192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:41:11.606336 13388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:11.587440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37825 (local address 127.0.98.1:35167)
0927 16:41:11.590079 (+ 2639us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:11.590135 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:11.591229 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:11.592989 (+ 1760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:11.592996 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:11.593316 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:41:11.593936 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:11.593955 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:11.595846 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:11.595849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:11.597090 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:11.597101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:11.598189 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:11.598192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:11.598353 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:11.598931 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:41:11.598949 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:41:11.600586 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:41:11.603038 (+ 2452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:11.603043 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:11.603056 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:11.603434 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:11.603857 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:11.603860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:11.603861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:11.604014 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:11.604721 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:41:11.604727 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:41:11.604992 (+ 265us) client_negotiation.cc:769] Sending connection context
0927 16:41:11.605165 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 16:41:11.605438 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2415,"thread_start_us":166,"threads_started":1}
I20260927 16:41:11.608084 13390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:11.588905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35167 (local address 127.0.98.62:37825)
0927 16:41:11.589782 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:11.589804 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:11.591512 (+ 1708us) server_negotiation.cc:407] Connection header received
0927 16:41:11.591796 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:11.591811 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:11.592175 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:11.592517 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:11.594116 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:11.595687 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:11.597220 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:11.597857 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:11.597995 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:11.600793 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:11.600799 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:11.600813 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:11.600841 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:11.602901 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:11.603577 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:11.603585 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:11.603591 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:11.603750 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:11.604122 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:11.604125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:11.604127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:11.604535 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:11.604667 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:11.606891 (+ 2224us) server_negotiation.cc:299] Negotiation successful
0927 16:41:11.607176 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":96,"threads_started":1}
I20260927 16:41:11.611261 13386 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37825
I20260927 16:41:11.611464 13386 heartbeater.cc:464] Registering TS with master...
I20260927 16:41:11.611967 13386 heartbeater.cc:511] Master 127.0.98.62:37825 requested a full tablet report, sending...
W20260927 16:41:11.614094 13386 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37825 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 16:41:11.614205 13386 heartbeater.cc:464] Registering TS with master...
I20260927 16:41:11.614351 13386 heartbeater.cc:511] Master 127.0.98.62:37825 requested a full tablet report, sending...
I20260927 16:41:11.615188 13386 heartbeater.cc:464] Registering TS with master...
I20260927 16:41:11.615398 13386 heartbeater.cc:511] Master 127.0.98.62:37825 requested a full tablet report, sending...
W20260927 16:41:11.616168 13386 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 16:41:11.738729 13389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:11.738965 13389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:11.739037 13389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:11.743083 13389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:11.743155 13389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:11.743275 13389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:41:11.750442 13389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55175/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44653
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33209
--tserver_master_addrs=127.0.98.62:37825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:11.751742 13389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:11.752915 13389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:41:11.763013 13396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:11.763300 13398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:11.763325 13389 server_base.cc:1034] running on GCE node
W20260927 16:41:11.763020 13395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:11.763803 13389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:11.764405 13389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:11.765558 13389 hybrid_clock.cc:648] HybridClock initialized: now 1790527271765530 us; error 44 us; skew 500 ppm
Sep 27 16:41:11 dist-test-slave-z40b krb5kdc[12192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:11.768227 13389 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:41:11.769115 13389 webserver.cc:460] Webserver started at http://127.0.98.2:33209/ using document root <none> and password file <none>
I20260927 16:41:11.769644 13389 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:11.769737 13389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 16:41:11 dist-test-slave-z40b krb5kdc[12192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:41:11.791170 13389 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 16:41:11.795959 13406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:11.797511 13389 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 16:41:11.797628 13389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "48eb0c04a8d7477f9f482420f3dd71a6"
format_stamp: "Formatted at 2026-09-27 16:41:10 on dist-test-slave-z40b"
server_key: "315d86b4d8f69e47539b141b3e6836d9"
server_key_iv: "1ab9ad4850e7e950b911c3f7e8c14dff"
server_key_version: "kuduclusterkey@0"
I20260927 16:41:11.798321 13389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:11.822363 13389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:11.825733 13389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:11.825934 13389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:11.826613 13389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:41:11.827680 13389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:41:11.827745 13389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:11.827811 13389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:41:11.827847 13389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:11.838577 13389 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44653
I20260927 16:41:11.838689 13519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44653 every 8 connection(s)
I20260927 16:41:11.839803 13389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:41:11.843762 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 13389
I20260927 16:41:11.844230 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55175/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39557
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37825
--webserver_port=45701
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55175/kms with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 16:41:11 dist-test-slave-z40b krb5kdc[12192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:41:11.860988 13390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:11.843441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42805 (local address 127.0.98.62:37825)
0927 16:41:11.843583 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:11.843588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:11.847693 (+ 4105us) server_negotiation.cc:407] Connection header received
0927 16:41:11.849057 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:11.849061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:11.849121 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:11.849222 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:11.850737 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:11.851607 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:11.853252 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:11.853622 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:11.853694 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:11.856784 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:11.856788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:11.856791 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:11.856819 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:11.858568 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:11.859185 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:11.859189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:11.859190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:11.859293 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:11.859740 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:11.859743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:11.859745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:11.859943 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:11.860019 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:11.860700 (+ 681us) server_negotiation.cc:299] Negotiation successful
0927 16:41:11.860822 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:41:11.862804 13522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:11.842954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37825 (local address 127.0.98.2:42805)
0927 16:41:11.847486 (+ 4532us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:11.847543 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:11.848794 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:11.849576 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:11.849585 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:11.849904 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:41:11.850562 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:11.850580 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:11.851760 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:11.851763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:11.853109 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:11.853119 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:11.853896 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:11.853899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:11.854082 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:11.854778 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:41:11.854797 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:41:11.856545 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:41:11.858729 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:11.858734 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:11.858748 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:11.859059 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:11.859400 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:11.859403 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:11.859405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:11.859630 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:11.860095 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:41:11.860100 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:41:11.860429 (+ 329us) client_negotiation.cc:769] Sending connection context
0927 16:41:11.861508 (+ 1079us) client_negotiation.cc:240] Negotiation successful
0927 16:41:11.861811 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4318,"thread_start_us":340,"threads_started":1}
I20260927 16:41:11.864279 13520 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37825
I20260927 16:41:11.864493 13520 heartbeater.cc:464] Registering TS with master...
I20260927 16:41:11.864988 13520 heartbeater.cc:511] Master 127.0.98.62:37825 requested a full tablet report, sending...
W20260927 16:41:11.866545 13520 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37825 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 16:41:11.866665 13520 heartbeater.cc:464] Registering TS with master...
I20260927 16:41:11.866803 13520 heartbeater.cc:511] Master 127.0.98.62:37825 requested a full tablet report, sending...
I20260927 16:41:11.867617 13520 heartbeater.cc:464] Registering TS with master...
I20260927 16:41:11.867760 13520 heartbeater.cc:511] Master 127.0.98.62:37825 requested a full tablet report, sending...
W20260927 16:41:11.868449 13520 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 16:41:12.003623 13523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:12.003942 13523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:12.004014 13523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:12.009975 13523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:12.010047 13523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:12.010131 13523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:41:12.017761 13523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55175/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45701
--tserver_master_addrs=127.0.98.62:37825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:12.019006 13523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:12.020360 13523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:41:12.032526 13529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:12.032753 13530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:12.035295 13523 server_base.cc:1034] running on GCE node
W20260927 16:41:12.039994 13532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:12.040496 13523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:12.041273 13523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:12.042441 13523 hybrid_clock.cc:648] HybridClock initialized: now 1790527272042407 us; error 47 us; skew 500 ppm
Sep 27 16:41:12 dist-test-slave-z40b krb5kdc[12192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:12.045444 13523 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:41:12.046435 13523 webserver.cc:460] Webserver started at http://127.0.98.3:45701/ using document root <none> and password file <none>
I20260927 16:41:12.046981 13523 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:12.047119 13523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 16:41:12 dist-test-slave-z40b krb5kdc[12192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:41:12.072058 13523 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:41:12.075014 13540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:12.076498 13523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:41:12.076606 13523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "7ad61091e523447d862a1f4447258ba8"
format_stamp: "Formatted at 2026-09-27 16:41:10 on dist-test-slave-z40b"
server_key: "00a912b48e476a28c5a441c229e7a7da"
server_key_iv: "9b37e53dfbe18e1c82412270a178e789"
server_key_version: "kuduclusterkey@0"
I20260927 16:41:12.077229 13523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:12.095933 13523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:12.099115 13523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:12.099320 13523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:12.099924 13523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:41:12.100888 13523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:41:12.100947 13523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:12.101006 13523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:41:12.101037 13523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:12.111946 13523 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39557
I20260927 16:41:12.113113 13523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:41:12.114849 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 13523
I20260927 16:41:12.115156 13653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39557 every 8 connection(s)
I20260927 16:41:12.125924 13390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:12.115583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52084 (local address 127.0.98.62:37825)
0927 16:41:12.115750 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:12.115755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:12.117327 (+ 1572us) server_negotiation.cc:407] Connection header received
0927 16:41:12.117451 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:12.117454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:12.117505 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:12.117614 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:12.118500 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:12.119287 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:12.120497 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:12.120852 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:12.120961 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:12.122573 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:12.122577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:12.122582 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:12.122622 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:12.124316 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:12.124723 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:12.124729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:12.124734 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:12.124824 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:12.125097 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:12.125102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:12.125104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:12.125330 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:12.125531 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:12.125654 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 16:41:12.125771 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 16:41:12 dist-test-slave-z40b krb5kdc[12192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:41:12.132046 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:12.117196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38909 (local address 127.0.98.62:37825)
0927 16:41:12.118501 (+ 1305us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:12.118510 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:12.118523 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:41:12.120498 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:12.120500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:12.120555 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:12.120652 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:12.121977 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:12.122765 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:12.124818 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:12.125172 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:12.125300 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:12.128342 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:12.128346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:12.128348 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:12.128371 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:12.129825 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:12.130471 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:12.130475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:12.130477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:12.130573 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:12.130940 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:12.130943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:12.130945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:12.131181 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:12.131316 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:12.131780 (+ 464us) server_negotiation.cc:299] Negotiation successful
0927 16:41:12.131892 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":99,"threads_started":1}
I20260927 16:41:12.132786 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:12.117731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37825 (local address 127.0.98.3:38909)
0927 16:41:12.118258 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:12.118312 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:12.120186 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:12.120830 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:12.120837 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:12.121158 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:41:12.121814 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:12.121836 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:12.122900 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:12.122903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:12.124333 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:12.124344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:12.125343 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:12.125347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:12.126070 (+ 723us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:12.126601 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:41:12.126616 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:41:12.128172 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:41:12.129946 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:12.129951 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:12.129964 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:12.130337 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:12.130688 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:12.130690 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:12.130691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:12.130819 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:12.131324 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:41:12.131329 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:41:12.131538 (+ 209us) client_negotiation.cc:769] Sending connection context
0927 16:41:12.131679 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 16:41:12.131936 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":137,"threads_started":1}
I20260927 16:41:12.133889 13654 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37825
I20260927 16:41:12.134083 13654 heartbeater.cc:464] Registering TS with master...
I20260927 16:41:12.134462 13654 heartbeater.cc:511] Master 127.0.98.62:37825 requested a full tablet report, sending...
W20260927 16:41:12.135931 13654 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37825 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 16:41:12.136029 13654 heartbeater.cc:464] Registering TS with master...
I20260927 16:41:12.136181 13654 heartbeater.cc:511] Master 127.0.98.62:37825 requested a full tablet report, sending...
I20260927 16:41:12.136857 13654 heartbeater.cc:464] Registering TS with master...
I20260927 16:41:12.136996 13654 heartbeater.cc:511] Master 127.0.98.62:37825 requested a full tablet report, sending...
W20260927 16:41:12.137614 13654 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 16:41:12 dist-test-slave-z40b krb5kdc[12192](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790527272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:12.616431 13386 heartbeater.cc:464] Registering TS with master...
I20260927 16:41:12.616765 13386 heartbeater.cc:511] Master 127.0.98.62:37825 requested a full tablet report, sending...
I20260927 16:41:12.617911 13188 ts_manager.cc:194] Registered new tserver with Master: 362b074721ca4dd7b81a62b40b601c3f (127.0.98.1:41715)
I20260927 16:41:12.619683 13188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35167
Sep 27 16:41:12 dist-test-slave-z40b krb5kdc[12192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:41:12.868643 13520 heartbeater.cc:464] Registering TS with master...
I20260927 16:41:12.868918 13520 heartbeater.cc:511] Master 127.0.98.62:37825 requested a full tablet report, sending...
I20260927 16:41:12.869668 13188 ts_manager.cc:194] Registered new tserver with Master: 48eb0c04a8d7477f9f482420f3dd71a6 (127.0.98.2:44653)
I20260927 16:41:12.870438 13188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42805
Sep 27 16:41:13 dist-test-slave-z40b krb5kdc[12192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 16:41:13.137830 13654 heartbeater.cc:464] Registering TS with master...
I20260927 16:41:13.138121 13654 heartbeater.cc:511] Master 127.0.98.62:37825 requested a full tablet report, sending...
I20260927 16:41:13.138846 13188 ts_manager.cc:194] Registered new tserver with Master: 7ad61091e523447d862a1f4447258ba8 (127.0.98.3:39557)
I20260927 16:41:13.139659 13188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38909
I20260927 16:41:13.143404 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:41:13.176215 13678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:13.148056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52100 (local address 127.0.98.62:37825)
0927 16:41:13.160011 (+ 11955us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:13.160024 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:13.160040 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:41:13.160114 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:13.160118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:13.160167 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:13.160271 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:13.168195 (+ 7924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:13.169042 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:13.170213 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:13.170569 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:13.170646 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:13.172391 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:13.172395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:13.172398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:13.172422 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:13.174062 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:13.174530 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:13.174534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:13.174535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:13.174620 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:13.174901 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:13.174904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:13.174906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:13.175123 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:13.175264 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:13.175920 (+ 656us) server_negotiation.cc:299] Negotiation successful
0927 16:41:13.176034 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11878,"thread_start_us":93,"threads_started":1}
I20260927 16:41:13.180084 13188 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:41:13.184904 13188 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:41:13.209115 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:13.198444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39557 (local address 127.0.0.1:52256)
0927 16:41:13.201220 (+ 2776us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:13.201234 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:13.202301 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:13.202995 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:13.203001 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:13.203025 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:41:13.203170 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:13.203178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:13.205342 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:13.205345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:13.206537 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:13.206544 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:13.208611 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:13.208614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:13.208723 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:13.208770 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 16:41:13.208869 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 16:41:13.208979 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2717,"thread_start_us":77,"threads_started":1}
I20260927 16:41:13.209928 13687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:13.198903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52256 (local address 127.0.98.3:39557)
0927 16:41:13.201072 (+ 2169us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:13.201079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:13.202383 (+ 1304us) server_negotiation.cc:407] Connection header received
0927 16:41:13.202463 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:13.202469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:13.202670 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:13.202813 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:41:13.203331 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:13.205222 (+ 1891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:13.206729 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:13.208310 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:13.208417 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:13.208547 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:13.209676 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0927 16:41:13.209761 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2064,"thread_start_us":82,"threads_started":1}
I20260927 16:41:13.213155 13588 tablet_service.cc:1467] Processing CreateTablet for tablet 5e9ca62423eb4beba4f672d6ee35c68c (DEFAULT_TABLE table=test-table [id=9a7e6d60886347eaa987be74050bc36e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:41:13.214109 13686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:13.197644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44653 (local address 127.0.0.1:50254)
0927 16:41:13.200623 (+ 2979us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:13.200674 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:13.200801 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:13.206538 (+ 5737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:13.206540 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:13.206557 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:41:13.206682 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:13.206689 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:13.209890 (+ 3201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:13.209910 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:13.211008 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:13.211017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:13.213750 (+ 2733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:13.213754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:13.213837 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:13.213857 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 16:41:13.213928 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 16:41:13.213991 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2805,"thread_start_us":82,"threads_started":1}
I20260927 16:41:13.214257 13588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e9ca62423eb4beba4f672d6ee35c68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:41:13.216208 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:13.197770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50254 (local address 127.0.98.2:44653)
0927 16:41:13.205832 (+ 8062us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:13.205838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:13.205860 (+ 22us) server_negotiation.cc:407] Connection header received
0927 16:41:13.205925 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:13.205929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:13.206085 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:13.206208 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:41:13.208781 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:13.209775 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:13.212451 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:13.213450 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:13.213550 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:13.213689 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:13.215963 (+ 2274us) server_negotiation.cc:299] Negotiation successful
0927 16:41:13.216071 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7965,"thread_start_us":50,"threads_started":1}
I20260927 16:41:13.218816 13454 tablet_service.cc:1467] Processing CreateTablet for tablet 5e9ca62423eb4beba4f672d6ee35c68c (DEFAULT_TABLE table=test-table [id=9a7e6d60886347eaa987be74050bc36e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:41:13.219781 13454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e9ca62423eb4beba4f672d6ee35c68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:41:13.240412 13690 tablet_bootstrap.cc:492] T 5e9ca62423eb4beba4f672d6ee35c68c P 48eb0c04a8d7477f9f482420f3dd71a6: Bootstrap starting.
I20260927 16:41:13.242877 13690 tablet_bootstrap.cc:654] T 5e9ca62423eb4beba4f672d6ee35c68c P 48eb0c04a8d7477f9f482420f3dd71a6: Neither blocks nor log segments found. Creating new log.
I20260927 16:41:13.246846 13688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:13.200148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41715 (local address 127.0.0.1:50192)
0927 16:41:13.202032 (+ 1884us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:13.202045 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:13.202153 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:13.238557 (+ 36404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:13.238561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:13.238580 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:41:13.238687 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:13.238693 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:13.243078 (+ 4385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:13.243083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:13.245288 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:13.245299 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:13.246444 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:13.246447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:13.246542 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:13.246571 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 16:41:13.246623 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 16:41:13.246700 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1827,"thread_start_us":78,"threads_started":1}
I20260927 16:41:13.247855 13689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:13.200294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50192 (local address 127.0.98.1:41715)
0927 16:41:13.238100 (+ 37806us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:13.238107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:13.238131 (+ 24us) server_negotiation.cc:407] Connection header received
0927 16:41:13.238218 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:13.238223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:13.238405 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:13.238834 (+ 429us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:41:13.238975 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:13.240005 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:13.245448 (+ 5443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:13.246302 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:13.247406 (+ 1104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:13.247562 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:13.247617 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 16:41:13.247713 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37700,"thread_start_us":121,"threads_started":1}
I20260927 16:41:13.248397 13691 tablet_bootstrap.cc:492] T 5e9ca62423eb4beba4f672d6ee35c68c P 7ad61091e523447d862a1f4447258ba8: Bootstrap starting.
I20260927 16:41:13.250795 13691 tablet_bootstrap.cc:654] T 5e9ca62423eb4beba4f672d6ee35c68c P 7ad61091e523447d862a1f4447258ba8: Neither blocks nor log segments found. Creating new log.
I20260927 16:41:13.252737 13690 log.cc:824] T 5e9ca62423eb4beba4f672d6ee35c68c P 48eb0c04a8d7477f9f482420f3dd71a6: Log is configured to *not* fsync() on all Append() calls
I20260927 16:41:13.254014 13691 log.cc:824] T 5e9ca62423eb4beba4f672d6ee35c68c P 7ad61091e523447d862a1f4447258ba8: Log is configured to *not* fsync() on all Append() calls
I20260927 16:41:13.256088 13302 tablet_service.cc:1467] Processing CreateTablet for tablet 5e9ca62423eb4beba4f672d6ee35c68c (DEFAULT_TABLE table=test-table [id=9a7e6d60886347eaa987be74050bc36e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:41:13.257198 13302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e9ca62423eb4beba4f672d6ee35c68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:41:13.257211 13690 tablet_bootstrap.cc:492] T 5e9ca62423eb4beba4f672d6ee35c68c P 48eb0c04a8d7477f9f482420f3dd71a6: No bootstrap required, opened a new log
I20260927 16:41:13.257452 13690 ts_tablet_manager.cc:1397] T 5e9ca62423eb4beba4f672d6ee35c68c P 48eb0c04a8d7477f9f482420f3dd71a6: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20260927 16:41:13.261545 13690 raft_consensus.cc:348] T 5e9ca62423eb4beba4f672d6ee35c68c P 48eb0c04a8d7477f9f482420f3dd71a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad61091e523447d862a1f4447258ba8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } } peers { permanent_uuid: "48eb0c04a8d7477f9f482420f3dd71a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "362b074721ca4dd7b81a62b40b601c3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41715 } }
I20260927 16:41:13.261907 13690 raft_consensus.cc:374] T 5e9ca62423eb4beba4f672d6ee35c68c P 48eb0c04a8d7477f9f482420f3dd71a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:41:13.262003 13690 raft_consensus.cc:729] T 5e9ca62423eb4beba4f672d6ee35c68c P 48eb0c04a8d7477f9f482420f3dd71a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48eb0c04a8d7477f9f482420f3dd71a6, State: Initialized, Role: FOLLOWER
I20260927 16:41:13.262549 13690 consensus_queue.cc:260] T 5e9ca62423eb4beba4f672d6ee35c68c P 48eb0c04a8d7477f9f482420f3dd71a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad61091e523447d862a1f4447258ba8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } } peers { permanent_uuid: "48eb0c04a8d7477f9f482420f3dd71a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "362b074721ca4dd7b81a62b40b601c3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41715 } }
I20260927 16:41:13.263731 13690 ts_tablet_manager.cc:1428] T 5e9ca62423eb4beba4f672d6ee35c68c P 48eb0c04a8d7477f9f482420f3dd71a6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 16:41:13.264521 13520 heartbeater.cc:503] Master 127.0.98.62:37825 was elected leader, sending a full tablet report...
I20260927 16:41:13.270054 13691 tablet_bootstrap.cc:492] T 5e9ca62423eb4beba4f672d6ee35c68c P 7ad61091e523447d862a1f4447258ba8: No bootstrap required, opened a new log
I20260927 16:41:13.270337 13691 ts_tablet_manager.cc:1397] T 5e9ca62423eb4beba4f672d6ee35c68c P 7ad61091e523447d862a1f4447258ba8: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20260927 16:41:13.274838 13691 raft_consensus.cc:348] T 5e9ca62423eb4beba4f672d6ee35c68c P 7ad61091e523447d862a1f4447258ba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad61091e523447d862a1f4447258ba8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } } peers { permanent_uuid: "48eb0c04a8d7477f9f482420f3dd71a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "362b074721ca4dd7b81a62b40b601c3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41715 } }
I20260927 16:41:13.275168 13691 raft_consensus.cc:374] T 5e9ca62423eb4beba4f672d6ee35c68c P 7ad61091e523447d862a1f4447258ba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:41:13.275307 13691 raft_consensus.cc:729] T 5e9ca62423eb4beba4f672d6ee35c68c P 7ad61091e523447d862a1f4447258ba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ad61091e523447d862a1f4447258ba8, State: Initialized, Role: FOLLOWER
I20260927 16:41:13.275841 13691 consensus_queue.cc:260] T 5e9ca62423eb4beba4f672d6ee35c68c P 7ad61091e523447d862a1f4447258ba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad61091e523447d862a1f4447258ba8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } } peers { permanent_uuid: "48eb0c04a8d7477f9f482420f3dd71a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "362b074721ca4dd7b81a62b40b601c3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41715 } }
I20260927 16:41:13.276918 13691 ts_tablet_manager.cc:1428] T 5e9ca62423eb4beba4f672d6ee35c68c P 7ad61091e523447d862a1f4447258ba8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 16:41:13.277284 13654 heartbeater.cc:503] Master 127.0.98.62:37825 was elected leader, sending a full tablet report...
I20260927 16:41:13.284708 13695 tablet_bootstrap.cc:492] T 5e9ca62423eb4beba4f672d6ee35c68c P 362b074721ca4dd7b81a62b40b601c3f: Bootstrap starting.
I20260927 16:41:13.286558 13695 tablet_bootstrap.cc:654] T 5e9ca62423eb4beba4f672d6ee35c68c P 362b074721ca4dd7b81a62b40b601c3f: Neither blocks nor log segments found. Creating new log.
I20260927 16:41:13.291896 13695 log.cc:824] T 5e9ca62423eb4beba4f672d6ee35c68c P 362b074721ca4dd7b81a62b40b601c3f: Log is configured to *not* fsync() on all Append() calls
I20260927 16:41:13.294171 13695 tablet_bootstrap.cc:492] T 5e9ca62423eb4beba4f672d6ee35c68c P 362b074721ca4dd7b81a62b40b601c3f: No bootstrap required, opened a new log
I20260927 16:41:13.294370 13695 ts_tablet_manager.cc:1397] T 5e9ca62423eb4beba4f672d6ee35c68c P 362b074721ca4dd7b81a62b40b601c3f: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20260927 16:41:13.297693 13695 raft_consensus.cc:348] T 5e9ca62423eb4beba4f672d6ee35c68c P 362b074721ca4dd7b81a62b40b601c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad61091e523447d862a1f4447258ba8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } } peers { permanent_uuid: "48eb0c04a8d7477f9f482420f3dd71a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "362b074721ca4dd7b81a62b40b601c3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41715 } }
I20260927 16:41:13.297966 13695 raft_consensus.cc:374] T 5e9ca62423eb4beba4f672d6ee35c68c P 362b074721ca4dd7b81a62b40b601c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:41:13.298035 13695 raft_consensus.cc:729] T 5e9ca62423eb4beba4f672d6ee35c68c P 362b074721ca4dd7b81a62b40b601c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 362b074721ca4dd7b81a62b40b601c3f, State: Initialized, Role: FOLLOWER
I20260927 16:41:13.298530 13695 consensus_queue.cc:260] T 5e9ca62423eb4beba4f672d6ee35c68c P 362b074721ca4dd7b81a62b40b601c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad61091e523447d862a1f4447258ba8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } } peers { permanent_uuid: "48eb0c04a8d7477f9f482420f3dd71a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "362b074721ca4dd7b81a62b40b601c3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41715 } }
I20260927 16:41:13.299219 13386 heartbeater.cc:503] Master 127.0.98.62:37825 was elected leader, sending a full tablet report...
I20260927 16:41:13.299592 13695 ts_tablet_manager.cc:1428] T 5e9ca62423eb4beba4f672d6ee35c68c P 362b074721ca4dd7b81a62b40b601c3f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260927 16:41:13.341972 13521 tablet.cc:2369] T 5e9ca62423eb4beba4f672d6ee35c68c P 48eb0c04a8d7477f9f482420f3dd71a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:41:13.344501 13387 tablet.cc:2369] T 5e9ca62423eb4beba4f672d6ee35c68c P 362b074721ca4dd7b81a62b40b601c3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:41:13.367996 13655 tablet.cc:2369] T 5e9ca62423eb4beba4f672d6ee35c68c P 7ad61091e523447d862a1f4447258ba8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:41:13.441998 13698 raft_consensus.cc:482] T 5e9ca62423eb4beba4f672d6ee35c68c P 362b074721ca4dd7b81a62b40b601c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:41:13.442224 13698 raft_consensus.cc:504] T 5e9ca62423eb4beba4f672d6ee35c68c P 362b074721ca4dd7b81a62b40b601c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad61091e523447d862a1f4447258ba8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } } peers { permanent_uuid: "48eb0c04a8d7477f9f482420f3dd71a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "362b074721ca4dd7b81a62b40b601c3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41715 } }
I20260927 16:41:13.443673 13698 leader_election.cc:288] T 5e9ca62423eb4beba4f672d6ee35c68c P 362b074721ca4dd7b81a62b40b601c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ad61091e523447d862a1f4447258ba8 (127.0.98.3:39557), 48eb0c04a8d7477f9f482420f3dd71a6 (127.0.98.2:44653)
I20260927 16:41:13.448136 13687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:13.444016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40045 (local address 127.0.98.3:39557)
0927 16:41:13.444155 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:13.444162 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:13.444470 (+ 308us) server_negotiation.cc:407] Connection header received
0927 16:41:13.444647 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:13.444651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:13.444703 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:13.444750 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:41:13.445153 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:13.445854 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:13.446930 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:13.447486 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:13.447592 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:13.447664 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:13.447919 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 16:41:13.448003 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:41:13.448154 13707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:13.443848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39557 (local address 127.0.98.1:40045)
0927 16:41:13.444378 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:13.444392 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:13.444500 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:13.444888 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:13.444890 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:13.444906 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:41:13.445023 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:13.445028 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:13.445969 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:13.445971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:13.446804 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:13.446813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:13.447624 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:13.447628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:13.447729 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:13.447755 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 16:41:13.447893 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 16:41:13.447977 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":292,"thread_start_us":101,"threads_started":1}
I20260927 16:41:13.448563 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:13.443987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44755 (local address 127.0.98.2:44653)
0927 16:41:13.444118 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:13.444123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:13.444419 (+ 296us) server_negotiation.cc:407] Connection header received
0927 16:41:13.444575 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:13.444580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:13.444661 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:13.444750 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:41:13.445030 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:13.445830 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:13.447314 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:13.447892 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:13.447989 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:13.448059 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:13.448395 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 16:41:13.448465 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:41:13.448578 13706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:13.443848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44653 (local address 127.0.98.1:44755)
0927 16:41:13.444315 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:13.444336 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:13.444461 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:13.444762 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:13.444765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:13.444780 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:41:13.444923 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:13.444927 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:13.445948 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:13.445951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:13.447162 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:13.447173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:13.448159 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:13.448164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:13.448265 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:13.448293 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 16:41:13.448380 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 16:41:13.448442 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":2,"thread_start_us":108,"threads_started":1}
I20260927 16:41:13.449120 13600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e9ca62423eb4beba4f672d6ee35c68c" candidate_uuid: "362b074721ca4dd7b81a62b40b601c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ad61091e523447d862a1f4447258ba8" is_pre_election: true
I20260927 16:41:13.449237 13474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e9ca62423eb4beba4f672d6ee35c68c" candidate_uuid: "362b074721ca4dd7b81a62b40b601c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48eb0c04a8d7477f9f482420f3dd71a6" is_pre_election: true
I20260927 16:41:13.449437 13600 raft_consensus.cc:2454] T 5e9ca62423eb4beba4f672d6ee35c68c P 7ad61091e523447d862a1f4447258ba8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 362b074721ca4dd7b81a62b40b601c3f in term 0.
I20260927 16:41:13.449518 13474 raft_consensus.cc:2454] T 5e9ca62423eb4beba4f672d6ee35c68c P 48eb0c04a8d7477f9f482420f3dd71a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 362b074721ca4dd7b81a62b40b601c3f in term 0.
I20260927 16:41:13.449851 13275 leader_election.cc:302] T 5e9ca62423eb4beba4f672d6ee35c68c P 362b074721ca4dd7b81a62b40b601c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 362b074721ca4dd7b81a62b40b601c3f, 7ad61091e523447d862a1f4447258ba8; no voters:
I20260927 16:41:13.450148 13698 raft_consensus.cc:2781] T 5e9ca62423eb4beba4f672d6ee35c68c P 362b074721ca4dd7b81a62b40b601c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:41:13.450229 13698 raft_consensus.cc:482] T 5e9ca62423eb4beba4f672d6ee35c68c P 362b074721ca4dd7b81a62b40b601c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:41:13.450273 13698 raft_consensus.cc:3037] T 5e9ca62423eb4beba4f672d6ee35c68c P 362b074721ca4dd7b81a62b40b601c3f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:41:13.451354 13698 raft_consensus.cc:504] T 5e9ca62423eb4beba4f672d6ee35c68c P 362b074721ca4dd7b81a62b40b601c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad61091e523447d862a1f4447258ba8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } } peers { permanent_uuid: "48eb0c04a8d7477f9f482420f3dd71a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "362b074721ca4dd7b81a62b40b601c3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41715 } }
I20260927 16:41:13.451710 13698 leader_election.cc:288] T 5e9ca62423eb4beba4f672d6ee35c68c P 362b074721ca4dd7b81a62b40b601c3f [CANDIDATE]: Term 1 election: Requested vote from peers 7ad61091e523447d862a1f4447258ba8 (127.0.98.3:39557), 48eb0c04a8d7477f9f482420f3dd71a6 (127.0.98.2:44653)
I20260927 16:41:13.451951 13600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e9ca62423eb4beba4f672d6ee35c68c" candidate_uuid: "362b074721ca4dd7b81a62b40b601c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ad61091e523447d862a1f4447258ba8"
I20260927 16:41:13.452037 13474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e9ca62423eb4beba4f672d6ee35c68c" candidate_uuid: "362b074721ca4dd7b81a62b40b601c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48eb0c04a8d7477f9f482420f3dd71a6"
I20260927 16:41:13.452085 13600 raft_consensus.cc:3037] T 5e9ca62423eb4beba4f672d6ee35c68c P 7ad61091e523447d862a1f4447258ba8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:41:13.452159 13474 raft_consensus.cc:3037] T 5e9ca62423eb4beba4f672d6ee35c68c P 48eb0c04a8d7477f9f482420f3dd71a6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:41:13.453054 13600 raft_consensus.cc:2454] T 5e9ca62423eb4beba4f672d6ee35c68c P 7ad61091e523447d862a1f4447258ba8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 362b074721ca4dd7b81a62b40b601c3f in term 1.
I20260927 16:41:13.453333 13275 leader_election.cc:302] T 5e9ca62423eb4beba4f672d6ee35c68c P 362b074721ca4dd7b81a62b40b601c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 362b074721ca4dd7b81a62b40b601c3f, 7ad61091e523447d862a1f4447258ba8; no voters:
I20260927 16:41:13.453454 13474 raft_consensus.cc:2454] T 5e9ca62423eb4beba4f672d6ee35c68c P 48eb0c04a8d7477f9f482420f3dd71a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 362b074721ca4dd7b81a62b40b601c3f in term 1.
I20260927 16:41:13.453492 13698 raft_consensus.cc:2781] T 5e9ca62423eb4beba4f672d6ee35c68c P 362b074721ca4dd7b81a62b40b601c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:41:13.453703 13698 raft_consensus.cc:686] T 5e9ca62423eb4beba4f672d6ee35c68c P 362b074721ca4dd7b81a62b40b601c3f [term 1 LEADER]: Becoming Leader. State: Replica: 362b074721ca4dd7b81a62b40b601c3f, State: Running, Role: LEADER
I20260927 16:41:13.454041 13698 consensus_queue.cc:237] T 5e9ca62423eb4beba4f672d6ee35c68c P 362b074721ca4dd7b81a62b40b601c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad61091e523447d862a1f4447258ba8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } } peers { permanent_uuid: "48eb0c04a8d7477f9f482420f3dd71a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "362b074721ca4dd7b81a62b40b601c3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41715 } }
I20260927 16:41:13.456875 13188 catalog_manager.cc:5495] T 5e9ca62423eb4beba4f672d6ee35c68c P 362b074721ca4dd7b81a62b40b601c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 362b074721ca4dd7b81a62b40b601c3f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "362b074721ca4dd7b81a62b40b601c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad61091e523447d862a1f4447258ba8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48eb0c04a8d7477f9f482420f3dd71a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "362b074721ca4dd7b81a62b40b601c3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41715 } health_report { overall_health: HEALTHY } } }
I20260927 16:41:13.505975 13689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:13.501985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50208 (local address 127.0.98.1:41715)
0927 16:41:13.502126 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:13.502131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:13.502147 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:41:13.502268 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:13.502271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:13.502334 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:13.502439 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:41:13.502655 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:13.503473 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:13.504423 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:13.504756 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:13.504851 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:13.505146 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:41:13.505640 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:41:13.505749 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:13.505778 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 16:41:13.505828 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 16:41:13.511952 13600 raft_consensus.cc:1261] T 5e9ca62423eb4beba4f672d6ee35c68c P 7ad61091e523447d862a1f4447258ba8 [term 1 FOLLOWER]: Refusing update from remote peer 362b074721ca4dd7b81a62b40b601c3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:41:13.511952 13474 raft_consensus.cc:1261] T 5e9ca62423eb4beba4f672d6ee35c68c P 48eb0c04a8d7477f9f482420f3dd71a6 [term 1 FOLLOWER]: Refusing update from remote peer 362b074721ca4dd7b81a62b40b601c3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:41:13.512979 13708 consensus_queue.cc:1035] T 5e9ca62423eb4beba4f672d6ee35c68c P 362b074721ca4dd7b81a62b40b601c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ad61091e523447d862a1f4447258ba8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:41:13.513188 13698 consensus_queue.cc:1035] T 5e9ca62423eb4beba4f672d6ee35c68c P 362b074721ca4dd7b81a62b40b601c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "48eb0c04a8d7477f9f482420f3dd71a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:41:13.527678 13713 mvcc.cc:205] Tried to move back new op lower bound from 7333999712296652800 to 7333999712070307840. Current Snapshot: MvccSnapshot[applied={T|T < 7333999712296652800}]
I20260927 16:41:13.532001 13188 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:41:13.532156 13188 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:41:13.534410 13188 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 741cb023496f4e66935d9d82d6e61f3a: Sending DeleteTablet for 3 replicas of tablet 5e9ca62423eb4beba4f672d6ee35c68c
I20260927 16:41:13.535166 13588 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e9ca62423eb4beba4f672d6ee35c68c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:41:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52256
I20260927 16:41:13.535384 13302 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e9ca62423eb4beba4f672d6ee35c68c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:41:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50192
I20260927 16:41:13.535774 13721 tablet_replica.cc:330] T 5e9ca62423eb4beba4f672d6ee35c68c P 7ad61091e523447d862a1f4447258ba8: stopping tablet replica
I20260927 16:41:13.535838 13454 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e9ca62423eb4beba4f672d6ee35c68c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:41:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50254
I20260927 16:41:13.536064 13721 raft_consensus.cc:2229] T 5e9ca62423eb4beba4f672d6ee35c68c P 7ad61091e523447d862a1f4447258ba8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:41:13.536320 13721 raft_consensus.cc:2258] T 5e9ca62423eb4beba4f672d6ee35c68c P 7ad61091e523447d862a1f4447258ba8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:41:13.537223 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 13226
I20260927 16:41:13.537369 13723 tablet_replica.cc:330] T 5e9ca62423eb4beba4f672d6ee35c68c P 48eb0c04a8d7477f9f482420f3dd71a6: stopping tablet replica
I20260927 16:41:13.538065 13723 raft_consensus.cc:2229] T 5e9ca62423eb4beba4f672d6ee35c68c P 48eb0c04a8d7477f9f482420f3dd71a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:41:13.538357 13723 raft_consensus.cc:2258] T 5e9ca62423eb4beba4f672d6ee35c68c P 48eb0c04a8d7477f9f482420f3dd71a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:41:13.540810 13723 ts_tablet_manager.cc:1905] T 5e9ca62423eb4beba4f672d6ee35c68c P 48eb0c04a8d7477f9f482420f3dd71a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:41:13.544158 13723 ts_tablet_manager.cc:1918] T 5e9ca62423eb4beba4f672d6ee35c68c P 48eb0c04a8d7477f9f482420f3dd71a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:41:13.544173 13721 ts_tablet_manager.cc:1905] T 5e9ca62423eb4beba4f672d6ee35c68c P 7ad61091e523447d862a1f4447258ba8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:41:13.544327 13723 log.cc:1196] T 5e9ca62423eb4beba4f672d6ee35c68c P 48eb0c04a8d7477f9f482420f3dd71a6: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-1/wal/wals/5e9ca62423eb4beba4f672d6ee35c68c
I20260927 16:41:13.544719 13723 ts_tablet_manager.cc:1939] T 5e9ca62423eb4beba4f672d6ee35c68c P 48eb0c04a8d7477f9f482420f3dd71a6: Deleting consensus metadata
W20260927 16:41:13.545086 13172 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41715 (error 108)
I20260927 16:41:13.545306 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 13389
I20260927 16:41:13.546876 13721 ts_tablet_manager.cc:1918] T 5e9ca62423eb4beba4f672d6ee35c68c P 7ad61091e523447d862a1f4447258ba8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:41:13.546995 13721 log.cc:1196] T 5e9ca62423eb4beba4f672d6ee35c68c P 7ad61091e523447d862a1f4447258ba8: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790527121515808-392-0/minicluster-data/ts-2/wal/wals/5e9ca62423eb4beba4f672d6ee35c68c
I20260927 16:41:13.547376 13721 ts_tablet_manager.cc:1939] T 5e9ca62423eb4beba4f672d6ee35c68c P 7ad61091e523447d862a1f4447258ba8: Deleting consensus metadata
I20260927 16:41:13.548065 13688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:13.547696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41715 (local address 127.0.0.1:50212)
0927 16:41:13.547890 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:13.547962 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260927 16:41:13.548339 13172 catalog_manager.cc:4584] TS 362b074721ca4dd7b81a62b40b601c3f (127.0.98.1:41715): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5e9ca62423eb4beba4f672d6ee35c68c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41715: connect: Connection refused (error 111)
I20260927 16:41:13.548547 13174 catalog_manager.cc:4841] TS 7ad61091e523447d862a1f4447258ba8 (127.0.98.3:39557): tablet 5e9ca62423eb4beba4f672d6ee35c68c (table test-table [id=9a7e6d60886347eaa987be74050bc36e]) successfully deleted
I20260927 16:41:13.551276 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 13523
I20260927 16:41:13.552174 13688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:13.551466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44653 (local address 127.0.0.1:50264)
0927 16:41:13.552037 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:13.552094 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476}
I20260927 16:41:13.558197 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 13155
I20260927 16:41:13.566299 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 16:41:13.900786 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 16:41:13.900884 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 16:41:14.246552 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 16:41:14.246 UTC [12222] LOG: received smart shutdown request
2026-09-27 16:41:14.250 UTC [12222] LOG: background worker "logical replication launcher" (PID 12231) exited with exit code 1
2026-09-27 16:41:14.250 UTC [12226] LOG: shutting down
2026-09-27 16:41:14.410 UTC [12222] LOG: database system is shut down
2026-09-27T16:41:14Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46193 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:41:14 dist-test-slave-z40b krb5kdc[13733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:41:14 dist-test-slave-z40b krb5kdc[13733](info): set up 2 sockets
Sep 27 16:41:14 dist-test-slave-z40b krb5kdc[13733](info): commencing operation
krb5kdc: starting...
W20260927 16:41:15.539072 392 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:41:15 dist-test-slave-z40b krb5kdc[13733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:41:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:41:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:41:15.652014 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45517
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45517,127.0.98.61:40825,127.0.98.60:35569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:41:15.754483 13746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:15.754688 13746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:15.754722 13746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:15.758584 13746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:41:15.758630 13746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:15.758648 13746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:15.758663 13746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:41:15.758678 13746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:41:15.763635 13746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45517,127.0.98.61:40825,127.0.98.60:35569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:15.764588 13746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:15.765506 13746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:41:15.773399 13754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:15.773537 13746 server_base.cc:1034] running on GCE node
W20260927 16:41:15.773589 13752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:15.773655 13751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:15.774219 13746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:15.775120 13746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:15.776293 13746 hybrid_clock.cc:648] HybridClock initialized: now 1790527275776281 us; error 31 us; skew 500 ppm
Sep 27 16:41:15 dist-test-slave-z40b krb5kdc[13733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:15.778663 13746 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:41:15.779489 13746 webserver.cc:460] Webserver started at http://127.0.98.62:33061/ using document root <none> and password file <none>
I20260927 16:41:15.779991 13746 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:15.780076 13746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:41:15.780313 13746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:41:15.782177 13746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "ae4aef87ff314a3c82840365ab974347"
format_stamp: "Formatted at 2026-09-27 16:41:15 on dist-test-slave-z40b"
server_key: "c4811af4a5c1436b156642ed69a56e11"
server_key_iv: "6229ed189c50f323bf43bd5c6c6fd3b6"
server_key_version: "encryptionkey@0"
I20260927 16:41:15.782670 13746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "ae4aef87ff314a3c82840365ab974347"
format_stamp: "Formatted at 2026-09-27 16:41:15 on dist-test-slave-z40b"
server_key: "c4811af4a5c1436b156642ed69a56e11"
server_key_iv: "6229ed189c50f323bf43bd5c6c6fd3b6"
server_key_version: "encryptionkey@0"
I20260927 16:41:15.786144 13746 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:41:15.788363 13761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:15.789422 13746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:41:15.789551 13746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "ae4aef87ff314a3c82840365ab974347"
format_stamp: "Formatted at 2026-09-27 16:41:15 on dist-test-slave-z40b"
server_key: "c4811af4a5c1436b156642ed69a56e11"
server_key_iv: "6229ed189c50f323bf43bd5c6c6fd3b6"
server_key_version: "encryptionkey@0"
I20260927 16:41:15.789656 13746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:15.810626 13746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:15.813217 13746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:15.813407 13746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:15.819828 13746 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45517
I20260927 16:41:15.819841 13813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45517 every 8 connection(s)
I20260927 16:41:15.820686 13746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:41:15.823248 13814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:41:15.826388 13814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45517 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:41:15.827384 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 13746
I20260927 16:41:15.827494 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:41:15.827780 392 external_mini_cluster.cc:1428] Setting key eeab30de8feb69413f4c68c7438f443b
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 16:41:15 dist-test-slave-z40b krb5kdc[13733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:41:15.843147 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:15.828558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45517 (local address 127.0.0.1:33674)
0927 16:41:15.828972 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:15.829017 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:15.829822 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:15.830486 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:15.830491 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:15.830885 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:41:15.831117 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:15.831130 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:15.832318 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:15.832320 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:15.833040 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:15.833049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:15.833640 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:15.833644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:15.833783 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:15.834349 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:41:15.834363 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:41:15.838314 (+ 3951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:41:15.840375 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:15.840383 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:15.840391 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:15.840630 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:15.840993 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:15.840995 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:15.840996 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:15.841113 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:15.841868 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:41:15.841873 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:41:15.841991 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 16:41:15.842136 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 16:41:15.842375 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":4,"thread_start_us":117,"threads_started":1}
I20260927 16:41:15.843155 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:15.828558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.0.98.62:45517)
0927 16:41:15.829023 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:15.829036 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:15.829129 (+ 93us) server_negotiation.cc:407] Connection header received
0927 16:41:15.830120 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:15.830124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:15.830306 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:15.830486 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:15.831310 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:15.832193 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:15.833161 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:15.833497 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:15.833729 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:15.838467 (+ 4738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:15.838476 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:15.838491 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:15.838523 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:15.840260 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:15.840744 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:15.840753 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:15.840758 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:15.840895 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:15.841224 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:15.841229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:15.841232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:15.841738 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:15.841885 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:15.842256 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 16:41:15.842457 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":349,"spinlock_wait_cycles":15232,"thread_start_us":130,"threads_started":1}
I20260927 16:41:15.845548 13814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40825 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:41:15.846261 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:15.845966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40825 (local address 127.0.0.1:38748)
0927 16:41:15.846122 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:15.846194 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 16:41:15.846459 13762 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40825: connect: Connection refused (error 111)
I20260927 16:41:15.847337 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:15.847073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40825 (local address 127.0.0.1:38764)
0927 16:41:15.847243 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:15.847292 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260927 16:41:15.847471 13814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40825: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40825: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 16:41:15.854040 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40825
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45517,127.0.98.61:40825,127.0.98.60:35569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:41:15.896832 13814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40825 } attempt: 1
I20260927 16:41:15.897719 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:15.897392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40825 (local address 127.0.0.1:38780)
0927 16:41:15.897576 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:15.897637 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 16:41:15.898442 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:15.898218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40825 (local address 127.0.0.1:38790)
0927 16:41:15.898318 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:15.898395 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 16:41:15.898566 13814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40825: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40825: connect: Connection refused (error 111)
W20260927 16:41:15.955633 13821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:15.955894 13821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:15.955966 13821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:15.959448 13821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:41:15.959517 13821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:15.959551 13821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:15.959605 13821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:41:15.959640 13821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:41:15.964463 13821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45517,127.0.98.61:40825,127.0.98.60:35569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:15.965355 13821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:15.966243 13821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:41:15.966722 13814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40825 } attempt: 2
I20260927 16:41:15.967535 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:15.967254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40825 (local address 127.0.0.1:38798)
0927 16:41:15.967401 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:15.967466 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 16:41:15.968191 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:15.967988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40825 (local address 127.0.0.1:38802)
0927 16:41:15.968092 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:15.968136 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 16:41:15.968349 13814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40825: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40825: connect: Connection refused (error 111)
W20260927 16:41:15.973992 13827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:15.974023 13826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:15.974195 13821 server_base.cc:1034] running on GCE node
W20260927 16:41:15.974023 13829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:15.974685 13821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:15.975636 13821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:15.976778 13821 hybrid_clock.cc:648] HybridClock initialized: now 1790527275976753 us; error 36 us; skew 500 ppm
Sep 27 16:41:15 dist-test-slave-z40b krb5kdc[13733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:15.979149 13821 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 16:41:15.980002 13821 webserver.cc:460] Webserver started at http://127.0.98.61:46655/ using document root <none> and password file <none>
I20260927 16:41:15.980515 13821 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:15.980602 13821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:41:15.980813 13821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:41:15.982715 13821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-1/data/instance:
uuid: "07dbe70e56a84777a6fb557bdf2f529a"
format_stamp: "Formatted at 2026-09-27 16:41:15 on dist-test-slave-z40b"
server_key: "17f7d710a571ca71b6defc5eaa32ecb0"
server_key_iv: "26f89fd55259c437d7d5409ea24927e4"
server_key_version: "encryptionkey@0"
I20260927 16:41:15.983296 13821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-1/wal/instance:
uuid: "07dbe70e56a84777a6fb557bdf2f529a"
format_stamp: "Formatted at 2026-09-27 16:41:15 on dist-test-slave-z40b"
server_key: "17f7d710a571ca71b6defc5eaa32ecb0"
server_key_iv: "26f89fd55259c437d7d5409ea24927e4"
server_key_version: "encryptionkey@0"
I20260927 16:41:15.986513 13821 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 16:41:15.988736 13836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:15.989759 13821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:41:15.989884 13821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-1/wal
uuid: "07dbe70e56a84777a6fb557bdf2f529a"
format_stamp: "Formatted at 2026-09-27 16:41:15 on dist-test-slave-z40b"
server_key: "17f7d710a571ca71b6defc5eaa32ecb0"
server_key_iv: "26f89fd55259c437d7d5409ea24927e4"
server_key_version: "encryptionkey@0"
I20260927 16:41:15.989984 13821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:16.009608 13821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:16.012526 13821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:16.012713 13821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:16.019265 13821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40825
I20260927 16:41:16.019335 13888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40825 every 8 connection(s)
I20260927 16:41:16.020149 13821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-1/data/info.pb
I20260927 16:41:16.022725 13889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:41:16.026151 13889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45517 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:41:16.029263 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 13821
I20260927 16:41:16.029377 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-1/wal/instance
I20260927 16:41:16.029641 392 external_mini_cluster.cc:1428] Setting key 3dddfd3a8f5be05b9cf4d6748018c69a
Sep 27 16:41:16 dist-test-slave-z40b krb5kdc[13733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 16:41:16.040056 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.027915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33686 (local address 127.0.98.62:45517)
0927 16:41:16.028041 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:16.028045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:16.028725 (+ 680us) server_negotiation.cc:407] Connection header received
0927 16:41:16.029818 (+ 1093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:16.029820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:16.029872 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:16.029935 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:16.031534 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.032364 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.033393 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.033826 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.033942 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:16.036343 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:16.036348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:16.036351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:16.036380 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:16.037914 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.038318 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.038320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.038321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.038385 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.038662 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.038664 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.038665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.038812 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:16.038881 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:16.039818 (+ 937us) server_negotiation.cc:299] Negotiation successful
0927 16:41:16.039907 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260927 16:41:16.040277 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.028178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45517 (local address 127.0.0.1:33686)
0927 16:41:16.028582 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:16.028619 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:16.029585 (+ 966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:16.030137 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:16.030148 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:16.030642 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:41:16.031337 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:16.031362 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.032484 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.032486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:16.033275 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:16.033283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.033966 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.033969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:16.034095 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:16.034633 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:41:16.034648 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:41:16.036145 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:41:16.038013 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:16.038017 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:16.038026 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:16.038239 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:16.038458 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:16.038460 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:16.038461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:16.038569 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:16.038898 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:41:16.038902 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:41:16.039091 (+ 189us) client_negotiation.cc:769] Sending connection context
0927 16:41:16.039291 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 16:41:16.039533 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":121,"threads_started":1}
I20260927 16:41:16.041741 13889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40825 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:41:16 dist-test-slave-z40b krb5kdc[13733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 16:41:16.051468 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.042174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40825 (local address 127.0.0.1:38810)
0927 16:41:16.042275 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:16.042288 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:16.042352 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:16.043036 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:16.043038 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:16.043148 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:41:16.043269 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:16.043274 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.044735 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.044737 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:16.045211 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:16.045218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.046006 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.046008 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:16.046088 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:16.046531 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:41:16.046541 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:41:16.047965 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:41:16.049702 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:16.049704 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:16.049706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:16.049968 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:16.050242 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:16.050245 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:16.050246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:16.050336 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:16.050889 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:41:16.050923 (+ 34us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:41:16.051041 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 16:41:16.051159 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 16:41:16.051309 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":2}
I20260927 16:41:16.051599 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.042323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38810 (local address 127.0.98.61:40825)
0927 16:41:16.042651 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:16.042664 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:16.042685 (+ 21us) server_negotiation.cc:407] Connection header received
0927 16:41:16.042739 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:16.042746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:16.042915 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:16.043041 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:16.043395 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.044623 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.045316 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.045906 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.046052 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:16.048089 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:16.048096 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:16.048105 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:16.048125 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:16.049601 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.050069 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.050073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.050074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.050142 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.050475 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.050477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.050478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.050786 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:16.050909 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:16.051296 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 16:41:16.051413 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":210,"thread_start_us":123,"threads_started":1}
I20260927 16:41:16.053246 13889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:41:16.053853 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.053608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:42484)
0927 16:41:16.053745 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:16.053807 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 16:41:16.054020 13840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
I20260927 16:41:16.054728 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.054517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:42496)
0927 16:41:16.054618 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:16.054667 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 16:41:16.054879 13889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35569: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 16:41:16.055860 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35569
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45517,127.0.98.61:40825,127.0.98.60:35569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:41:16.059489 13814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40825 } attempt: 3
Sep 27 16:41:16 dist-test-slave-z40b krb5kdc[13733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 16:41:16.067945 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.060030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38816 (local address 127.0.98.61:40825)
0927 16:41:16.060143 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:16.060147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:16.060162 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:41:16.060206 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:16.060209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:16.060244 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:16.060361 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:16.060748 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.061243 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.062004 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.062741 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.062854 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:16.065112 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:16.065114 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:16.065116 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:16.065134 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:16.066452 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.066895 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.066897 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.066899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.066966 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.067284 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.067287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.067288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.067432 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:16.067504 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:16.067772 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 16:41:16.067851 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:41:16.068005 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.059900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40825 (local address 127.0.0.1:38816)
0927 16:41:16.060015 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:16.060030 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:16.060130 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:16.060375 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:16.060378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:16.060535 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:41:16.060626 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:16.060631 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.061358 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.061361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:16.061892 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:16.061899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.062867 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.062870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:16.062978 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:16.063460 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:41:16.063472 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:41:16.064975 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:41:16.066555 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:16.066558 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:16.066561 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:16.066804 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:16.067050 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:16.067052 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:16.067054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:16.067150 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:16.067520 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:41:16.067523 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:41:16.067615 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 16:41:16.067726 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 16:41:16.067846 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 16:41:16.068651 13814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:41:16.069273 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.069066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:42512)
0927 16:41:16.069169 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:16.069220 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 16:41:16.069939 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.069724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:42522)
0927 16:41:16.069855 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:16.069893 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260927 16:41:16.070112 13814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35569: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
I20260927 16:41:16.101250 13814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } attempt: 1
I20260927 16:41:16.101967 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.101716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:42538)
0927 16:41:16.101848 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:16.101900 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 16:41:16.102623 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.102428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:42542)
0927 16:41:16.102529 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:16.102575 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 16:41:16.102753 13814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35569: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
I20260927 16:41:16.104063 13889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } attempt: 1
I20260927 16:41:16.104661 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.104417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:42558)
0927 16:41:16.104548 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:16.104603 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 16:41:16.105306 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.105102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:42562)
0927 16:41:16.105214 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:16.105261 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 16:41:16.105422 13889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35569: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
W20260927 16:41:16.162115 13896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:16.162370 13896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:16.162446 13896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:16.166298 13896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:41:16.166388 13896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:16.166424 13896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:16.166482 13896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:41:16.166517 13896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:41:16.171530 13896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45517,127.0.98.61:40825,127.0.98.60:35569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:16.172427 13896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:16.173342 13896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:41:16.173581 13889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } attempt: 2
I20260927 16:41:16.174306 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.174037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:42576)
0927 16:41:16.174166 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:16.174238 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 16:41:16.174961 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.174774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:42584)
0927 16:41:16.174872 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:16.174912 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 16:41:16.175091 13889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35569: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
I20260927 16:41:16.177901 13814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } attempt: 2
I20260927 16:41:16.178484 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.178264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:42586)
0927 16:41:16.178380 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:16.178426 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 16:41:16.179121 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.178949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:42600)
0927 16:41:16.179036 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:16.179075 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 16:41:16.179329 13814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35569: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
W20260927 16:41:16.181696 13904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:16.181764 13902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:16.181759 13901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:16.181946 13896 server_base.cc:1034] running on GCE node
I20260927 16:41:16.182399 13896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:16.183398 13896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:16.184556 13896 hybrid_clock.cc:648] HybridClock initialized: now 1790527276184541 us; error 23 us; skew 500 ppm
Sep 27 16:41:16 dist-test-slave-z40b krb5kdc[13733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:16.187104 13896 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 16:41:16.187952 13896 webserver.cc:460] Webserver started at http://127.0.98.60:41815/ using document root <none> and password file <none>
I20260927 16:41:16.188428 13896 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:16.188481 13896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:41:16.188623 13896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:41:16.190424 13896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-2/data/instance:
uuid: "3bf9fd5236af4889b82dc14482391c81"
format_stamp: "Formatted at 2026-09-27 16:41:16 on dist-test-slave-z40b"
server_key: "c72b23db757c247929863866c1e1ec95"
server_key_iv: "31c7274edcd063113cf67cc00f266283"
server_key_version: "encryptionkey@0"
I20260927 16:41:16.190887 13896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-2/wal/instance:
uuid: "3bf9fd5236af4889b82dc14482391c81"
format_stamp: "Formatted at 2026-09-27 16:41:16 on dist-test-slave-z40b"
server_key: "c72b23db757c247929863866c1e1ec95"
server_key_iv: "31c7274edcd063113cf67cc00f266283"
server_key_version: "encryptionkey@0"
I20260927 16:41:16.194411 13896 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:41:16.196527 13911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:16.197613 13896 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:41:16.197741 13896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-2/wal
uuid: "3bf9fd5236af4889b82dc14482391c81"
format_stamp: "Formatted at 2026-09-27 16:41:16 on dist-test-slave-z40b"
server_key: "c72b23db757c247929863866c1e1ec95"
server_key_iv: "31c7274edcd063113cf67cc00f266283"
server_key_version: "encryptionkey@0"
I20260927 16:41:16.197839 13896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:16.213681 13896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:16.216104 13896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:16.216290 13896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:16.222780 13896 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35569
I20260927 16:41:16.222792 13963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35569 every 8 connection(s)
I20260927 16:41:16.223696 13896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-2/data/info.pb
I20260927 16:41:16.226133 13964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:41:16.229583 13964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45517 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:41:16.231057 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 13896
I20260927 16:41:16.231191 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790527121515808-392-0/minicluster-data/master-2/wal/instance
I20260927 16:41:16.231505 392 external_mini_cluster.cc:1428] Setting key ed0109f15f560e5303ac124cebcbc6bf
Sep 27 16:41:16 dist-test-slave-z40b krb5kdc[13733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 16:41:16 dist-test-slave-z40b krb5kdc[13733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 16:41:16.242288 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.231426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33702 (local address 127.0.98.62:45517)
0927 16:41:16.231591 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:16.231597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:16.232513 (+ 916us) server_negotiation.cc:407] Connection header received
0927 16:41:16.233362 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:16.233365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:16.233427 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:16.233513 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:16.234237 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.234701 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.235629 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.235829 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.235931 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:16.238331 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:16.238334 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:16.238336 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:16.238360 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:16.240110 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.240694 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.240699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.240702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.240790 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.241157 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.241160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.241162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.241404 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:16.241476 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:16.241994 (+ 518us) server_negotiation.cc:299] Negotiation successful
0927 16:41:16.242129 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 16:41:16.242974 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.231836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45517 (local address 127.0.0.1:33702)
0927 16:41:16.232337 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:16.232399 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:16.233172 (+ 773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:16.233557 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:16.233562 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:16.233897 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:41:16.234104 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:16.234114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.234802 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.234807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:16.235517 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:16.235526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.235959 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.235963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:16.236133 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:16.236684 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:41:16.236707 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:41:16.238164 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:41:16.240216 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:16.240223 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:16.240230 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:16.240591 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:16.240895 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:16.240898 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:16.240900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:16.241064 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:16.241506 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:41:16.241514 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:41:16.241743 (+ 229us) client_negotiation.cc:769] Sending connection context
0927 16:41:16.241918 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 16:41:16.242208 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":126,"threads_started":1}
I20260927 16:41:16.242985 13968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.232143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42604 (local address 127.0.98.60:35569)
0927 16:41:16.232800 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:16.232809 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:16.232837 (+ 28us) server_negotiation.cc:407] Connection header received
0927 16:41:16.233188 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:16.233206 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:16.233371 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:16.233543 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:16.234282 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.235004 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.235804 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.236128 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.236246 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:16.238815 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:16.238837 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:16.238851 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:16.238875 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:16.240680 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.241173 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.241178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.241180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.241252 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.241531 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.241534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.241536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.241759 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:16.241928 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:16.242166 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 16:41:16.242288 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":98,"threads_started":1}
I20260927 16:41:16.244158 13964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40825 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:41:16 dist-test-slave-z40b krb5kdc[13733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 16:41:16 dist-test-slave-z40b krb5kdc[13733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:41:16.252169 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.244580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40825 (local address 127.0.0.1:38820)
0927 16:41:16.244841 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:16.244852 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:16.244913 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:16.245122 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:16.245123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:16.245260 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:41:16.245374 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:16.245381 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.246047 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.246049 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:16.246830 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:16.246843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.247400 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.247403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:16.247473 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:16.247912 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:41:16.247923 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:41:16.249090 (+ 1167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:41:16.250748 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:16.250754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:16.250759 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:16.250947 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:16.251254 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:16.251257 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:16.251259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:16.251349 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:16.251718 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:41:16.251720 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:41:16.251809 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 16:41:16.251912 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 16:41:16.252023 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192}
I20260927 16:41:16.252205 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.244710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38820 (local address 127.0.98.61:40825)
0927 16:41:16.244840 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:16.244845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:16.244889 (+ 44us) server_negotiation.cc:407] Connection header received
0927 16:41:16.244997 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:16.245000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:16.245040 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:16.245097 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:16.245488 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.245936 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.246943 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.247302 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.247402 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:16.249230 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:16.249237 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:16.249241 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:16.249271 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:16.250622 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.251045 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.251049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.251051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.251134 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.251445 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.251447 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.251448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.251613 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:16.251725 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:16.251962 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 16:41:16.252064 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:41:16.252892 13964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:41:16.254573 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.245405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33712 (local address 127.0.98.62:45517)
0927 16:41:16.246093 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:16.246096 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:16.246105 (+ 9us) server_negotiation.cc:407] Connection header received
0927 16:41:16.246130 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:16.246132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:16.246171 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:16.246226 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:16.246886 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.247589 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.248351 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.248622 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.248727 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:16.251282 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:16.251288 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:16.251294 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:16.251321 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:16.252957 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.253430 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.253435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.253439 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.253521 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.253818 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.253821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.253823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.254023 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:16.254247 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:16.254391 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 16:41:16.254474 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626}
Sep 27 16:41:16 dist-test-slave-z40b krb5kdc[13733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 16:41:16 dist-test-slave-z40b krb5kdc[13733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 16:41:16.261041 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.253326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:42606)
0927 16:41:16.253429 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:16.253441 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:16.253508 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:16.253826 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:16.253830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:16.253979 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:41:16.254063 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:16.254071 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.255117 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.255120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:16.255839 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:16.255848 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.256260 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.256263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:16.256388 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:16.256840 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:41:16.256855 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:41:16.258029 (+ 1174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:41:16.259594 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:16.259598 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:16.259602 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:16.259869 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:16.260118 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:16.260124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:16.260127 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:16.260231 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:16.260649 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:41:16.260651 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:41:16.260719 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 16:41:16.260796 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 16:41:16.260920 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":15}
I20260927 16:41:16.261318 13968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.253493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42606 (local address 127.0.98.60:35569)
0927 16:41:16.253602 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:16.253606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:16.253616 (+ 10us) server_negotiation.cc:407] Connection header received
0927 16:41:16.253645 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:16.253648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:16.253697 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:16.253793 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:16.254165 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.255003 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.255922 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.256157 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.256263 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:16.258139 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:16.258143 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:16.258145 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:16.258174 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:16.259488 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.259965 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.259967 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.259968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.260030 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.260346 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.260350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.260354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.260553 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:16.260654 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:16.260880 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 16:41:16.260987 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":41}
I20260927 16:41:16.265544 13964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bf9fd5236af4889b82dc14482391c81: Bootstrap starting.
I20260927 16:41:16.265820 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.255686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38826 (local address 127.0.98.61:40825)
0927 16:41:16.255774 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:16.255778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:16.255790 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:41:16.255828 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:16.255830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:16.255858 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:16.255943 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:16.256508 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.256973 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.257984 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.258301 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.258405 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:16.261387 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:16.261392 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:16.261396 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:16.261428 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:16.263796 (+ 2368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.264245 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.264248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.264250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.264395 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.264727 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.264730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.264732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.265019 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:16.265273 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:16.265553 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 16:41:16.265656 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 16:41:16.266247 13889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } attempt: 3
I20260927 16:41:16.268549 13964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bf9fd5236af4889b82dc14482391c81: Neither blocks nor log segments found. Creating new log.
I20260927 16:41:16.268785 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:41:16.269338 13964 log.cc:824] T 00000000000000000000000000000000 P 3bf9fd5236af4889b82dc14482391c81: Log is configured to *not* fsync() on all Append() calls
Sep 27 16:41:16 dist-test-slave-z40b krb5kdc[13733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 16:41:16.271876 13964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bf9fd5236af4889b82dc14482391c81: No bootstrap required, opened a new log
I20260927 16:41:16.275158 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.266574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:42622)
0927 16:41:16.266664 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:16.266677 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:16.266743 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:16.266956 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:16.266959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:16.267169 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:41:16.267312 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:16.267322 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.268190 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.268201 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:16.268941 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:16.268952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.269542 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.269545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:16.269624 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:16.270099 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:41:16.270112 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:41:16.271568 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:41:16.273602 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:16.273605 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:16.273608 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:16.273863 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:16.274215 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:16.274218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:16.274219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:16.274302 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:16.274748 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:41:16.274751 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:41:16.274830 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 16:41:16.274916 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 16:41:16.275021 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 16:41:16.275684 13968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.266633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42622 (local address 127.0.98.60:35569)
0927 16:41:16.266735 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:16.266738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:16.266749 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:41:16.266812 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:16.266815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:16.266853 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:16.266924 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:16.267437 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.268074 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.269060 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.269401 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.269540 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:16.271738 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:16.271744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:16.271750 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:16.271779 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:16.273473 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.273976 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.273980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.273981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.274063 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.274396 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.274401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.274404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.274613 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:16.274708 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:16.275446 (+ 738us) server_negotiation.cc:299] Negotiation successful
0927 16:41:16.275544 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 16:41:16.276036 13964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bf9fd5236af4889b82dc14482391c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4aef87ff314a3c82840365ab974347" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45517 } } peers { permanent_uuid: "07dbe70e56a84777a6fb557bdf2f529a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40825 } } peers { permanent_uuid: "3bf9fd5236af4889b82dc14482391c81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } }
I20260927 16:41:16.276286 13964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bf9fd5236af4889b82dc14482391c81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:41:16.276345 13964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bf9fd5236af4889b82dc14482391c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bf9fd5236af4889b82dc14482391c81, State: Initialized, Role: FOLLOWER
I20260927 16:41:16.276871 13964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bf9fd5236af4889b82dc14482391c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4aef87ff314a3c82840365ab974347" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45517 } } peers { permanent_uuid: "07dbe70e56a84777a6fb557bdf2f529a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40825 } } peers { permanent_uuid: "3bf9fd5236af4889b82dc14482391c81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } }
I20260927 16:41:16.278149 13964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bf9fd5236af4889b82dc14482391c81 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:41:16.278458 13814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } attempt: 3
I20260927 16:41:16.279055 13889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07dbe70e56a84777a6fb557bdf2f529a: Bootstrap starting.
I20260927 16:41:16.283761 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.271108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38838 (local address 127.0.98.61:40825)
0927 16:41:16.271222 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:16.271226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:16.271856 (+ 630us) server_negotiation.cc:407] Connection header received
0927 16:41:16.272084 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:16.272087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:16.272124 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:16.272175 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:16.273049 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.273764 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.274959 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.275299 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.275368 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:16.280021 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:16.280024 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:16.280027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:16.280053 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:16.281649 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.282437 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.282440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.282441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.282529 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.282843 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.282857 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.282858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.283029 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:16.283139 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:16.283482 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 16:41:16.283577 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 16:41:16.283855 13989 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3bf9fd5236af4889b82dc14482391c81: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
Sep 27 16:41:16 dist-test-slave-z40b krb5kdc[13733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
W20260927 16:41:16.283921 13989 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:41:16.284703 13978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bf9fd5236af4889b82dc14482391c81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4aef87ff314a3c82840365ab974347" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45517 } } peers { permanent_uuid: "07dbe70e56a84777a6fb557bdf2f529a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40825 } } peers { permanent_uuid: "3bf9fd5236af4889b82dc14482391c81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } } }
I20260927 16:41:16.286069 13978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bf9fd5236af4889b82dc14482391c81 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:41:16.282382 13896 master_runner.cc:186] Error getting master registration for 127.0.98.62:45517: OK, instance_id { permanent_uuid: "ae4aef87ff314a3c82840365ab974347" instance_seqno: 1790527275818015 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 16:41:16.288419 13896 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 16:41:16.288769 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.271511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33718 (local address 127.0.98.62:45517)
0927 16:41:16.271630 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:16.271633 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:16.272649 (+ 1016us) server_negotiation.cc:407] Connection header received
0927 16:41:16.272829 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:16.272834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:16.272886 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:16.272944 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:16.274508 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.275183 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.276919 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.277204 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.277270 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:16.285020 (+ 7750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:16.285024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:16.285026 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:16.285050 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:16.286688 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.287356 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.287360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.287361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.287440 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.287726 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.287730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.287731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.287925 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:16.288027 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:16.288534 (+ 507us) server_negotiation.cc:299] Negotiation successful
0927 16:41:16.288634 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:41:16.288959 13889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07dbe70e56a84777a6fb557bdf2f529a: Neither blocks nor log segments found. Creating new log.
I20260927 16:41:16.289064 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.278772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:42642)
0927 16:41:16.278964 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:16.278977 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:16.279042 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:16.279514 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:16.279516 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:16.279679 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:41:16.279759 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:16.279765 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.280713 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.280716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:16.281485 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:16.281496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.282325 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.282328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:16.282413 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:16.282835 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:41:16.282859 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:41:16.284253 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:41:16.287536 (+ 3283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:16.287540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:16.287542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:16.287787 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:16.288093 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:16.288096 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:16.288098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:16.288184 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:16.288621 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:41:16.288624 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:41:16.288709 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 16:41:16.288788 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 16:41:16.288913 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260927 16:41:16.289539 13968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.279172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42642 (local address 127.0.98.60:35569)
0927 16:41:16.279279 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:16.279283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:16.279294 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:41:16.279322 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:16.279324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:16.279357 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:16.279423 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:16.279878 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.280590 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.281736 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.282101 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.282228 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:16.284392 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:16.284395 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:16.284398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:16.284421 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:16.287401 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.287905 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.287909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.287911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.287983 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.288310 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.288312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.288314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.288519 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:16.289232 (+ 713us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:16.289302 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 16:41:16.289397 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":57}
I20260927 16:41:16.289913 13889 log.cc:824] T 00000000000000000000000000000000 P 07dbe70e56a84777a6fb557bdf2f529a: Log is configured to *not* fsync() on all Append() calls
I20260927 16:41:16.290220 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:16.271889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42634 (local address 127.0.98.60:35569)
0927 16:41:16.272423 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:16.272432 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:16.274218 (+ 1786us) server_negotiation.cc:407] Connection header received
0927 16:41:16.274249 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:16.274254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:16.274295 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:16.274368 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:16.277384 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.278125 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.279737 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:16.281907 (+ 2170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:16.281993 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:16.285747 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:16.285751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:16.285753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:16.285778 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:16.287627 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.288104 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.288107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.288109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.288184 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:16.288553 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:16.288559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:16.288564 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:16.288739 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:16.288856 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:16.289903 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0927 16:41:16.290023 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":954,"server-negotiator.queue_time_us":466,"thread_start_us":86,"threads_started":1}
I20260927 16:41:16.292215 13889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07dbe70e56a84777a6fb557bdf2f529a: No bootstrap required, opened a new log
I20260927 16:41:16.292436 13814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347: Bootstrap starting.
I20260927 16:41:16.294827 13814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347: Neither blocks nor log segments found. Creating new log.
I20260927 16:41:16.296088 13889 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07dbe70e56a84777a6fb557bdf2f529a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4aef87ff314a3c82840365ab974347" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45517 } } peers { permanent_uuid: "07dbe70e56a84777a6fb557bdf2f529a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40825 } } peers { permanent_uuid: "3bf9fd5236af4889b82dc14482391c81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } }
I20260927 16:41:16.296288 13889 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07dbe70e56a84777a6fb557bdf2f529a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:41:16.296303 13814 log.cc:824] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347: Log is configured to *not* fsync() on all Append() calls
I20260927 16:41:16.296340 13889 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07dbe70e56a84777a6fb557bdf2f529a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07dbe70e56a84777a6fb557bdf2f529a, State: Initialized, Role: FOLLOWER
I20260927 16:41:16.296769 13889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07dbe70e56a84777a6fb557bdf2f529a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4aef87ff314a3c82840365ab974347" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45517 } } peers { permanent_uuid: "07dbe70e56a84777a6fb557bdf2f529a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40825 } } peers { permanent_uuid: "3bf9fd5236af4889b82dc14482391c81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } }
I20260927 16:41:16.297935 13889 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07dbe70e56a84777a6fb557bdf2f529a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:41:16.298183 13814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347: No bootstrap required, opened a new log
I20260927 16:41:16.299492 13993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07dbe70e56a84777a6fb557bdf2f529a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4aef87ff314a3c82840365ab974347" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45517 } } peers { permanent_uuid: "07dbe70e56a84777a6fb557bdf2f529a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40825 } } peers { permanent_uuid: "3bf9fd5236af4889b82dc14482391c81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } } }
I20260927 16:41:16.299677 13993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07dbe70e56a84777a6fb557bdf2f529a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:41:16.302371 13814 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4aef87ff314a3c82840365ab974347" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45517 } } peers { permanent_uuid: "07dbe70e56a84777a6fb557bdf2f529a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40825 } } peers { permanent_uuid: "3bf9fd5236af4889b82dc14482391c81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } }
I20260927 16:41:16.302592 13814 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:41:16.302654 13814 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4aef87ff314a3c82840365ab974347, State: Initialized, Role: FOLLOWER
I20260927 16:41:16.303809 13814 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4aef87ff314a3c82840365ab974347" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45517 } } peers { permanent_uuid: "07dbe70e56a84777a6fb557bdf2f529a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40825 } } peers { permanent_uuid: "3bf9fd5236af4889b82dc14482391c81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } }
I20260927 16:41:16.304306 14005 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4aef87ff314a3c82840365ab974347" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45517 } } peers { permanent_uuid: "07dbe70e56a84777a6fb557bdf2f529a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40825 } } peers { permanent_uuid: "3bf9fd5236af4889b82dc14482391c81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } } }
I20260927 16:41:16.304587 14005 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:41:16.300973 13821 master_runner.cc:186] Error getting master registration for 127.0.98.62:45517: OK, instance_id { permanent_uuid: "ae4aef87ff314a3c82840365ab974347" instance_seqno: 1790527275818015 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 16:41:16.304821 13821 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 16:41:16.305063 13814 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 16:41:16.307395 14004 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 07dbe70e56a84777a6fb557bdf2f529a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:41:16.307466 14004 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 16:41:16.312208 14017 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:41:16.312282 14017 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:41:16.313113 13746 master_runner.cc:263] No leader master found from master ae4aef87ff314a3c82840365ab974347
I20260927 16:41:16.313202 13746 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 16:41:16.314759 13767 tablet.cc:2369] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:41:16.463052 14005 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:41:16.463260 14005 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4aef87ff314a3c82840365ab974347" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45517 } } peers { permanent_uuid: "07dbe70e56a84777a6fb557bdf2f529a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40825 } } peers { permanent_uuid: "3bf9fd5236af4889b82dc14482391c81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } }
I20260927 16:41:16.464141 14005 leader_election.cc:288] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07dbe70e56a84777a6fb557bdf2f529a (127.0.98.61:40825), 3bf9fd5236af4889b82dc14482391c81 (127.0.98.60:35569)
I20260927 16:41:16.464614 13863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae4aef87ff314a3c82840365ab974347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07dbe70e56a84777a6fb557bdf2f529a" is_pre_election: true
I20260927 16:41:16.464908 13863 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 07dbe70e56a84777a6fb557bdf2f529a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae4aef87ff314a3c82840365ab974347 in term 0.
I20260927 16:41:16.465036 13938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae4aef87ff314a3c82840365ab974347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bf9fd5236af4889b82dc14482391c81" is_pre_election: true
I20260927 16:41:16.465269 13938 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3bf9fd5236af4889b82dc14482391c81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae4aef87ff314a3c82840365ab974347 in term 0.
I20260927 16:41:16.465646 13765 leader_election.cc:302] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bf9fd5236af4889b82dc14482391c81, ae4aef87ff314a3c82840365ab974347; no voters:
I20260927 16:41:16.465914 14005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:41:16.465993 14005 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:41:16.466043 14005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:41:16.466996 14005 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4aef87ff314a3c82840365ab974347" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45517 } } peers { permanent_uuid: "07dbe70e56a84777a6fb557bdf2f529a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40825 } } peers { permanent_uuid: "3bf9fd5236af4889b82dc14482391c81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } }
I20260927 16:41:16.467433 14005 leader_election.cc:288] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [CANDIDATE]: Term 1 election: Requested vote from peers 07dbe70e56a84777a6fb557bdf2f529a (127.0.98.61:40825), 3bf9fd5236af4889b82dc14482391c81 (127.0.98.60:35569)
W20260927 16:41:16.467460 13917 tablet.cc:2369] T 00000000000000000000000000000000 P 3bf9fd5236af4889b82dc14482391c81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:41:16.468044 13938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae4aef87ff314a3c82840365ab974347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bf9fd5236af4889b82dc14482391c81"
I20260927 16:41:16.468164 13938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bf9fd5236af4889b82dc14482391c81 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:41:16.469213 13938 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3bf9fd5236af4889b82dc14482391c81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae4aef87ff314a3c82840365ab974347 in term 1.
I20260927 16:41:16.469511 13765 leader_election.cc:302] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bf9fd5236af4889b82dc14482391c81, ae4aef87ff314a3c82840365ab974347; no voters:
I20260927 16:41:16.469735 14005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:41:16.469707 13863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae4aef87ff314a3c82840365ab974347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07dbe70e56a84777a6fb557bdf2f529a"
I20260927 16:41:16.469816 13863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07dbe70e56a84777a6fb557bdf2f529a [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:41:16.469942 14005 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [term 1 LEADER]: Becoming Leader. State: Replica: ae4aef87ff314a3c82840365ab974347, State: Running, Role: LEADER
I20260927 16:41:16.470408 14005 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4aef87ff314a3c82840365ab974347" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45517 } } peers { permanent_uuid: "07dbe70e56a84777a6fb557bdf2f529a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40825 } } peers { permanent_uuid: "3bf9fd5236af4889b82dc14482391c81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } }
I20260927 16:41:16.471056 13863 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 07dbe70e56a84777a6fb557bdf2f529a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae4aef87ff314a3c82840365ab974347 in term 1.
I20260927 16:41:16.472215 14019 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae4aef87ff314a3c82840365ab974347. Latest consensus state: current_term: 1 leader_uuid: "ae4aef87ff314a3c82840365ab974347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4aef87ff314a3c82840365ab974347" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45517 } } peers { permanent_uuid: "07dbe70e56a84777a6fb557bdf2f529a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40825 } } peers { permanent_uuid: "3bf9fd5236af4889b82dc14482391c81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } } }
I20260927 16:41:16.472335 14019 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [sys.catalog]: This master's current role is: LEADER
I20260927 16:41:16.472656 14021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:41:16.473646 14021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:41:16.477799 13863 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 07dbe70e56a84777a6fb557bdf2f529a [term 1 FOLLOWER]: Refusing update from remote peer ae4aef87ff314a3c82840365ab974347: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:41:16.477833 13938 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3bf9fd5236af4889b82dc14482391c81 [term 1 FOLLOWER]: Refusing update from remote peer ae4aef87ff314a3c82840365ab974347: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:41:16.478754 14005 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07dbe70e56a84777a6fb557bdf2f529a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:41:16.479133 14005 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bf9fd5236af4889b82dc14482391c81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:41:16.484318 13978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bf9fd5236af4889b82dc14482391c81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae4aef87ff314a3c82840365ab974347. Latest consensus state: current_term: 1 leader_uuid: "ae4aef87ff314a3c82840365ab974347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4aef87ff314a3c82840365ab974347" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45517 } } peers { permanent_uuid: "07dbe70e56a84777a6fb557bdf2f529a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40825 } } peers { permanent_uuid: "3bf9fd5236af4889b82dc14482391c81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } } }
I20260927 16:41:16.484429 13978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bf9fd5236af4889b82dc14482391c81 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:41:16.485381 13993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07dbe70e56a84777a6fb557bdf2f529a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae4aef87ff314a3c82840365ab974347. Latest consensus state: current_term: 1 leader_uuid: "ae4aef87ff314a3c82840365ab974347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4aef87ff314a3c82840365ab974347" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45517 } } peers { permanent_uuid: "07dbe70e56a84777a6fb557bdf2f529a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40825 } } peers { permanent_uuid: "3bf9fd5236af4889b82dc14482391c81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } } }
I20260927 16:41:16.485483 13993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07dbe70e56a84777a6fb557bdf2f529a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:41:16.487576 14005 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae4aef87ff314a3c82840365ab974347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4aef87ff314a3c82840365ab974347" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45517 } } peers { permanent_uuid: "07dbe70e56a84777a6fb557bdf2f529a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40825 } } peers { permanent_uuid: "3bf9fd5236af4889b82dc14482391c81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } } }
I20260927 16:41:16.487680 14005 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [sys.catalog]: This master's current role is: LEADER
I20260927 16:41:16.488137 13978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bf9fd5236af4889b82dc14482391c81 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae4aef87ff314a3c82840365ab974347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4aef87ff314a3c82840365ab974347" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45517 } } peers { permanent_uuid: "07dbe70e56a84777a6fb557bdf2f529a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40825 } } peers { permanent_uuid: "3bf9fd5236af4889b82dc14482391c81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } } }
I20260927 16:41:16.488236 13978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bf9fd5236af4889b82dc14482391c81 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:41:16.489297 14025 mvcc.cc:205] Tried to move back new op lower bound from 7333999724447133696 to 7333999724426346496. Current Snapshot: MvccSnapshot[applied={T|T < 7333999724447133696 or (T in {7333999724447133696})}]
I20260927 16:41:16.491194 14021 catalog_manager.cc:1347] Generated new cluster ID: cbc3daab05b84c4f8a855713b279d14c
I20260927 16:41:16.491269 14021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:41:16.498353 14005 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae4aef87ff314a3c82840365ab974347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4aef87ff314a3c82840365ab974347" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45517 } } peers { permanent_uuid: "07dbe70e56a84777a6fb557bdf2f529a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40825 } } peers { permanent_uuid: "3bf9fd5236af4889b82dc14482391c81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } } }
I20260927 16:41:16.498466 14005 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347 [sys.catalog]: This master's current role is: LEADER
I20260927 16:41:16.499261 14023 mvcc.cc:205] Tried to move back new op lower bound from 7333999724447133696 to 7333999724426346496. Current Snapshot: MvccSnapshot[applied={T|T < 7333999724447133696}]
I20260927 16:41:16.499652 13993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07dbe70e56a84777a6fb557bdf2f529a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae4aef87ff314a3c82840365ab974347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4aef87ff314a3c82840365ab974347" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45517 } } peers { permanent_uuid: "07dbe70e56a84777a6fb557bdf2f529a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40825 } } peers { permanent_uuid: "3bf9fd5236af4889b82dc14482391c81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } } }
I20260927 16:41:16.499761 13993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07dbe70e56a84777a6fb557bdf2f529a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:41:16.506820 14021 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:41:16.508014 14021 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:41:16.520156 14021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae4aef87ff314a3c82840365ab974347: Generated new TSK 0
I20260927 16:41:16.520903 14021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:41:17.285157 13989 catalog_manager.cc:1259] Loaded cluster ID: cbc3daab05b84c4f8a855713b279d14c
I20260927 16:41:17.285249 13989 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3bf9fd5236af4889b82dc14482391c81: loading cluster ID for follower catalog manager: success
I20260927 16:41:17.286933 13989 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3bf9fd5236af4889b82dc14482391c81: acquiring CA information for follower catalog manager: success
I20260927 16:41:17.288144 13989 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3bf9fd5236af4889b82dc14482391c81: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 16:41:17.308552 14004 catalog_manager.cc:1259] Loaded cluster ID: cbc3daab05b84c4f8a855713b279d14c
I20260927 16:41:17.308635 14004 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 07dbe70e56a84777a6fb557bdf2f529a: loading cluster ID for follower catalog manager: success
I20260927 16:41:17.310209 14004 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 07dbe70e56a84777a6fb557bdf2f529a: acquiring CA information for follower catalog manager: success
I20260927 16:41:17.311336 14004 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 07dbe70e56a84777a6fb557bdf2f529a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 16:41:17.576478 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 13746
I20260927 16:41:17.582969 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 13821
I20260927 16:41:17.588622 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 13896
2026-09-27T16:41:17Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3060 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:41:17 dist-test-slave-z40b krb5kdc[14038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:41:17 dist-test-slave-z40b krb5kdc[14038](info): set up 2 sockets
Sep 27 16:41:17 dist-test-slave-z40b krb5kdc[14038](info): commencing operation
krb5kdc: starting...
W20260927 16:41:18.607474 392 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:41:18 dist-test-slave-z40b krb5kdc[14038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:41:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:41:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:41:18.736261 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37841
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:41:18.841480 14051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:18.841701 14051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:18.841776 14051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:18.845709 14051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:41:18.845757 14051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:18.845775 14051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:18.845829 14051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:41:18.845857 14051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:41:18.850885 14051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:18.851891 14051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:18.852840 14051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:41:18.861027 14059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:18.861017 14056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:18.861002 14057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:18.861277 14051 server_base.cc:1034] running on GCE node
I20260927 16:41:18.861878 14051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:18.862884 14051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:18.864046 14051 hybrid_clock.cc:648] HybridClock initialized: now 1790527278864032 us; error 27 us; skew 500 ppm
Sep 27 16:41:18 dist-test-slave-z40b krb5kdc[14038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:18.866601 14051 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:41:18.867592 14051 webserver.cc:460] Webserver started at http://127.0.98.62:40243/ using document root <none> and password file <none>
I20260927 16:41:18.868110 14051 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:18.868223 14051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:41:18.868458 14051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:41:18.870432 14051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "8710da4e9a9b4a839f42ffb751dda277"
format_stamp: "Formatted at 2026-09-27 16:41:18 on dist-test-slave-z40b"
server_key: "d78e47616da008deb33f4344e7e42be8"
server_key_iv: "726332e377e78890e11e2a20d8f56791"
server_key_version: "encryptionkey@0"
I20260927 16:41:18.870962 14051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "8710da4e9a9b4a839f42ffb751dda277"
format_stamp: "Formatted at 2026-09-27 16:41:18 on dist-test-slave-z40b"
server_key: "d78e47616da008deb33f4344e7e42be8"
server_key_iv: "726332e377e78890e11e2a20d8f56791"
server_key_version: "encryptionkey@0"
I20260927 16:41:18.874377 14051 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:41:18.876705 14066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:18.877779 14051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:41:18.877915 14051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "8710da4e9a9b4a839f42ffb751dda277"
format_stamp: "Formatted at 2026-09-27 16:41:18 on dist-test-slave-z40b"
server_key: "d78e47616da008deb33f4344e7e42be8"
server_key_iv: "726332e377e78890e11e2a20d8f56791"
server_key_version: "encryptionkey@0"
I20260927 16:41:18.878026 14051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:18.909144 14051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:18.911940 14051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:18.912125 14051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:18.918869 14051 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37841
I20260927 16:41:18.918892 14118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37841 every 8 connection(s)
I20260927 16:41:18.919756 14051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:41:18.921435 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 14051
I20260927 16:41:18.921562 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:41:18.921883 392 external_mini_cluster.cc:1428] Setting key fda46d4b478a22f49915696ecdce01c2
I20260927 16:41:18.922576 14119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:41:18.927971 14119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8710da4e9a9b4a839f42ffb751dda277: Bootstrap starting.
Sep 27 16:41:18 dist-test-slave-z40b krb5kdc[14038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:41:18.930365 14119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8710da4e9a9b4a839f42ffb751dda277: Neither blocks nor log segments found. Creating new log.
I20260927 16:41:18.931203 14119 log.cc:824] T 00000000000000000000000000000000 P 8710da4e9a9b4a839f42ffb751dda277: Log is configured to *not* fsync() on all Append() calls
I20260927 16:41:18.933171 14119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8710da4e9a9b4a839f42ffb751dda277: No bootstrap required, opened a new log
I20260927 16:41:18.935160 14122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:18.923139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59006 (local address 127.0.98.62:37841)
0927 16:41:18.923573 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:18.923586 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:18.923619 (+ 33us) server_negotiation.cc:407] Connection header received
0927 16:41:18.924242 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:18.924256 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:18.924551 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:18.924876 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:18.925697 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:18.926645 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:18.927579 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:18.928053 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:18.928230 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:18.930895 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:18.930900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:18.930911 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:18.930936 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:18.932586 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:18.933062 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:18.933066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:18.933070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:18.933170 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:18.933459 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:18.933461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:18.933462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:18.933801 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:18.933931 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:18.934224 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 16:41:18.934510 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":144,"threads_started":1}
I20260927 16:41:18.937057 14119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8710da4e9a9b4a839f42ffb751dda277 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8710da4e9a9b4a839f42ffb751dda277" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37841 } }
I20260927 16:41:18.937340 14119 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8710da4e9a9b4a839f42ffb751dda277 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:41:18.937431 14119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8710da4e9a9b4a839f42ffb751dda277 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8710da4e9a9b4a839f42ffb751dda277, State: Initialized, Role: FOLLOWER
I20260927 16:41:18.937937 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:41:18.937901 14119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8710da4e9a9b4a839f42ffb751dda277 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8710da4e9a9b4a839f42ffb751dda277" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37841 } }
I20260927 16:41:18.938046 14119 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8710da4e9a9b4a839f42ffb751dda277 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:41:18.938107 14119 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8710da4e9a9b4a839f42ffb751dda277 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:41:18.938190 14119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8710da4e9a9b4a839f42ffb751dda277 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:41:18.939198 14119 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8710da4e9a9b4a839f42ffb751dda277 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8710da4e9a9b4a839f42ffb751dda277" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37841 } }
I20260927 16:41:18.939554 14119 leader_election.cc:302] T 00000000000000000000000000000000 P 8710da4e9a9b4a839f42ffb751dda277 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8710da4e9a9b4a839f42ffb751dda277; no voters:
I20260927 16:41:18.939828 14119 leader_election.cc:288] T 00000000000000000000000000000000 P 8710da4e9a9b4a839f42ffb751dda277 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:41:18.939940 14124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8710da4e9a9b4a839f42ffb751dda277 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:41:18.940162 14124 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8710da4e9a9b4a839f42ffb751dda277 [term 1 LEADER]: Becoming Leader. State: Replica: 8710da4e9a9b4a839f42ffb751dda277, State: Running, Role: LEADER
I20260927 16:41:18.940471 14124 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8710da4e9a9b4a839f42ffb751dda277 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8710da4e9a9b4a839f42ffb751dda277" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37841 } }
I20260927 16:41:18.940919 14119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8710da4e9a9b4a839f42ffb751dda277 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:41:18.942072 14126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8710da4e9a9b4a839f42ffb751dda277 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8710da4e9a9b4a839f42ffb751dda277. Latest consensus state: current_term: 1 leader_uuid: "8710da4e9a9b4a839f42ffb751dda277" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8710da4e9a9b4a839f42ffb751dda277" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37841 } } }
I20260927 16:41:18.942200 14126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8710da4e9a9b4a839f42ffb751dda277 [sys.catalog]: This master's current role is: LEADER
I20260927 16:41:18.942492 14125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8710da4e9a9b4a839f42ffb751dda277 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8710da4e9a9b4a839f42ffb751dda277" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8710da4e9a9b4a839f42ffb751dda277" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37841 } } }
I20260927 16:41:18.942602 14125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8710da4e9a9b4a839f42ffb751dda277 [sys.catalog]: This master's current role is: LEADER
I20260927 16:41:18.942660 14133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:41:18.945605 14133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:41:18.950747 14133 catalog_manager.cc:1347] Generated new cluster ID: ecde3290502a41b48b88ceebce905ba9
I20260927 16:41:18.950814 14133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:41:18.960812 14133 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:41:18.961614 14133 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:41:18.967870 14133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8710da4e9a9b4a839f42ffb751dda277: Generated new TSK 0
I20260927 16:41:18.968458 14133 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:41:18.993690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 14051
2026-09-27T16:41:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1403 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:41:19 dist-test-slave-z40b krb5kdc[14146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:41:19 dist-test-slave-z40b krb5kdc[14146](info): set up 2 sockets
Sep 27 16:41:19 dist-test-slave-z40b krb5kdc[14146](info): commencing operation
krb5kdc: starting...
W20260927 16:41:20.015415 392 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:41:20 dist-test-slave-z40b krb5kdc[14146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:41:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:41:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:41:20.126183 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45119
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45119
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:41:20.230497 14159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:20.230760 14159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:20.230841 14159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:20.234791 14159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:41:20.234875 14159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:20.234913 14159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:20.234969 14159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:41:20.235006 14159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:41:20.239984 14159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:20.241031 14159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:20.242271 14159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:41:20.250489 14167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:20.250515 14164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:20.250564 14165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:20.250633 14159 server_base.cc:1034] running on GCE node
I20260927 16:41:20.251380 14159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:20.252339 14159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:20.253504 14159 hybrid_clock.cc:648] HybridClock initialized: now 1790527280253486 us; error 28 us; skew 500 ppm
Sep 27 16:41:20 dist-test-slave-z40b krb5kdc[14146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:20.262446 14159 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:41:20.263388 14159 webserver.cc:460] Webserver started at http://127.0.98.62:39371/ using document root <none> and password file <none>
I20260927 16:41:20.263893 14159 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:20.263949 14159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:41:20.264114 14159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:41:20.265974 14159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "ed559add211b4fe190913711a83ace58"
format_stamp: "Formatted at 2026-09-27 16:41:20 on dist-test-slave-z40b"
server_key: "7dcc356e390aaab0e210d29dd3a38588"
server_key_iv: "543efba9c1852c62580d11adf0a57e14"
server_key_version: "encryptionkey@0"
I20260927 16:41:20.266436 14159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "ed559add211b4fe190913711a83ace58"
format_stamp: "Formatted at 2026-09-27 16:41:20 on dist-test-slave-z40b"
server_key: "7dcc356e390aaab0e210d29dd3a38588"
server_key_iv: "543efba9c1852c62580d11adf0a57e14"
server_key_version: "encryptionkey@0"
I20260927 16:41:20.269930 14159 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 16:41:20.272209 14174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:20.273324 14159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:41:20.273416 14159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "ed559add211b4fe190913711a83ace58"
format_stamp: "Formatted at 2026-09-27 16:41:20 on dist-test-slave-z40b"
server_key: "7dcc356e390aaab0e210d29dd3a38588"
server_key_iv: "543efba9c1852c62580d11adf0a57e14"
server_key_version: "encryptionkey@0"
I20260927 16:41:20.273491 14159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:20.295357 14159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:20.298374 14159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:20.298513 14159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:20.305418 14159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45119
I20260927 16:41:20.305699 14226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45119 every 8 connection(s)
I20260927 16:41:20.306356 14159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:41:20.308917 14227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:41:20.311457 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 14159
I20260927 16:41:20.311589 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:41:20.311857 392 external_mini_cluster.cc:1428] Setting key 57e61f441320809ac83af8b7f989afa2
I20260927 16:41:20.314502 14227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58: Bootstrap starting.
I20260927 16:41:20.316943 14227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58: Neither blocks nor log segments found. Creating new log.
I20260927 16:41:20.317751 14227 log.cc:824] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58: Log is configured to *not* fsync() on all Append() calls
I20260927 16:41:20.319945 14227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58: No bootstrap required, opened a new log
Sep 27 16:41:20 dist-test-slave-z40b krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:41:20.323805 14227 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed559add211b4fe190913711a83ace58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } }
I20260927 16:41:20.324041 14227 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:41:20.324098 14227 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed559add211b4fe190913711a83ace58, State: Initialized, Role: FOLLOWER
I20260927 16:41:20.324538 14227 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed559add211b4fe190913711a83ace58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } }
I20260927 16:41:20.324678 14227 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:41:20.324745 14227 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:41:20.324836 14227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:41:20.325698 14227 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed559add211b4fe190913711a83ace58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } }
I20260927 16:41:20.326012 14227 leader_election.cc:302] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed559add211b4fe190913711a83ace58; no voters:
I20260927 16:41:20.326292 14227 leader_election.cc:288] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:41:20.326324 14230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:20.313172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46172 (local address 127.0.98.62:45119)
0927 16:41:20.313557 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:20.313572 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:20.313604 (+ 32us) server_negotiation.cc:407] Connection header received
0927 16:41:20.314206 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:20.314223 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:20.314525 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:20.314842 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:20.315728 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:20.316692 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:20.317872 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:20.318512 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:20.318628 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:20.321218 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:20.321224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:20.321238 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:20.321267 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:20.323720 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:20.324115 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:20.324123 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:20.324133 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:20.324252 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:20.324531 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:20.324536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:20.324538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:20.324883 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:20.325097 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:20.325372 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 16:41:20.325623 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":124,"threads_started":1}
I20260927 16:41:20.326422 14232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:41:20.326625 14232 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [term 1 LEADER]: Becoming Leader. State: Replica: ed559add211b4fe190913711a83ace58, State: Running, Role: LEADER
I20260927 16:41:20.326913 14232 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed559add211b4fe190913711a83ace58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } }
I20260927 16:41:20.327507 14227 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:41:20.328051 14234 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed559add211b4fe190913711a83ace58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed559add211b4fe190913711a83ace58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } } }
I20260927 16:41:20.328166 14234 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [sys.catalog]: This master's current role is: LEADER
I20260927 16:41:20.328648 14237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:41:20.328933 14233 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed559add211b4fe190913711a83ace58. Latest consensus state: current_term: 1 leader_uuid: "ed559add211b4fe190913711a83ace58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed559add211b4fe190913711a83ace58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } } }
I20260927 16:41:20.329037 14233 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [sys.catalog]: This master's current role is: LEADER
I20260927 16:41:20.331674 14237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:41:20.337123 14237 catalog_manager.cc:1347] Generated new cluster ID: 06701aaea5784b5385b3207bbd6a9839
I20260927 16:41:20.337201 14237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:41:20.360778 14237 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:41:20.361706 14237 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:41:20.373200 14237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58: Generated new TSK 0
I20260927 16:41:20.373936 14237 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:41:20.407950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45119
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:41:20.513808 14253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:20.514021 14253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:20.514082 14253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:20.518038 14253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:20.518087 14253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:20.518165 14253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:41:20.523054 14253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:20.523983 14253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:20.524886 14253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:41:20.533811 14261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:20.533880 14259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:20.534034 14253 server_base.cc:1034] running on GCE node
W20260927 16:41:20.533927 14258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:20.534477 14253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:20.535065 14253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:20.536237 14253 hybrid_clock.cc:648] HybridClock initialized: now 1790527280536211 us; error 45 us; skew 500 ppm
Sep 27 16:41:20 dist-test-slave-z40b krb5kdc[14146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:20.538738 14253 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:41:20.539530 14253 webserver.cc:460] Webserver started at http://127.0.98.1:33135/ using document root <none> and password file <none>
I20260927 16:41:20.540045 14253 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:20.540129 14253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:41:20.540344 14253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:41:20.542164 14253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/data/instance:
uuid: "bbff7cfa13fc4c1ab07b892db9b7f105"
format_stamp: "Formatted at 2026-09-27 16:41:20 on dist-test-slave-z40b"
server_key: "e017c70c57309ce7d4c9835237eee7e6"
server_key_iv: "abe5007fb5b487bc4768842fe1a91804"
server_key_version: "encryptionkey@0"
I20260927 16:41:20.542717 14253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bbff7cfa13fc4c1ab07b892db9b7f105"
format_stamp: "Formatted at 2026-09-27 16:41:20 on dist-test-slave-z40b"
server_key: "e017c70c57309ce7d4c9835237eee7e6"
server_key_iv: "abe5007fb5b487bc4768842fe1a91804"
server_key_version: "encryptionkey@0"
I20260927 16:41:20.546093 14253 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:41:20.548288 14268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:20.549328 14253 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:41:20.549451 14253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "bbff7cfa13fc4c1ab07b892db9b7f105"
format_stamp: "Formatted at 2026-09-27 16:41:20 on dist-test-slave-z40b"
server_key: "e017c70c57309ce7d4c9835237eee7e6"
server_key_iv: "abe5007fb5b487bc4768842fe1a91804"
server_key_version: "encryptionkey@0"
I20260927 16:41:20.549547 14253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:20.571420 14253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:20.574031 14253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:20.574218 14253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:20.574733 14253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:41:20.575522 14253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:41:20.575590 14253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:20.575656 14253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:41:20.575706 14253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:20.584613 14253 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33767
I20260927 16:41:20.584635 14381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33767 every 8 connection(s)
I20260927 16:41:20.585398 14253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:41:20.593577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 14253
I20260927 16:41:20.593714 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:41:20.593994 392 external_mini_cluster.cc:1428] Setting key ca3ded267d1ab6cdfee3a9781dc4cdcc
Sep 27 16:41:20 dist-test-slave-z40b krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:41:20.598497 14230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:20.587009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40457 (local address 127.0.98.62:45119)
0927 16:41:20.587153 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:20.587157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:20.588403 (+ 1246us) server_negotiation.cc:407] Connection header received
0927 16:41:20.589180 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:20.589183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:20.589247 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:20.589319 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:20.590299 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:20.590816 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:20.592037 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:20.592251 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:20.592331 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:20.594755 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:20.594758 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:20.594760 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:20.594782 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:20.596426 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:20.596874 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:20.596880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:20.596885 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:20.596974 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:20.597491 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:20.597494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:20.597495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:20.597647 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:20.597727 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:20.598276 (+ 549us) server_negotiation.cc:299] Negotiation successful
0927 16:41:20.598391 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:41:20.599293 14384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:20.587457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:40457)
0927 16:41:20.588264 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:20.588303 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:20.589015 (+ 712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:20.589435 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:20.589440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:20.589750 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:41:20.590175 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:20.590186 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:20.590920 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:20.590922 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:20.591935 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:20.591944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:20.592373 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:20.592375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:20.592499 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:20.592947 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:41:20.592962 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:41:20.594582 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:41:20.596537 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:20.596541 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:20.596551 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:20.596779 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:20.597174 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:20.597181 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:20.597194 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:20.597395 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:20.597746 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:41:20.597756 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:41:20.598036 (+ 280us) client_negotiation.cc:769] Sending connection context
0927 16:41:20.598251 (+ 215us) client_negotiation.cc:240] Negotiation successful
0927 16:41:20.598489 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":515,"threads_started":1}
I20260927 16:41:20.600615 14382 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45119
I20260927 16:41:20.600839 14382 heartbeater.cc:464] Registering TS with master...
I20260927 16:41:20.601259 14382 heartbeater.cc:511] Master 127.0.98.62:45119 requested a full tablet report, sending...
I20260927 16:41:20.602650 14191 ts_manager.cc:194] Registered new tserver with Master: bbff7cfa13fc4c1ab07b892db9b7f105 (127.0.98.1:33767)
I20260927 16:41:20.603773 14191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40457
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:41:20.621488 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45119
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:41:20.726154 14387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:20.726375 14387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:20.726413 14387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:20.730446 14387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:20.730494 14387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:20.730575 14387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:41:20.735430 14387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:20.736241 14387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:20.737161 14387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:41:20.745891 14393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:20.745966 14387 server_base.cc:1034] running on GCE node
W20260927 16:41:20.745916 14392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:20.745920 14395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:20.746606 14387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:20.747125 14387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:20.748344 14387 hybrid_clock.cc:648] HybridClock initialized: now 1790527280748318 us; error 39 us; skew 500 ppm
Sep 27 16:41:20 dist-test-slave-z40b krb5kdc[14146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:20.750765 14387 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:41:20.751598 14387 webserver.cc:460] Webserver started at http://127.0.98.2:39323/ using document root <none> and password file <none>
I20260927 16:41:20.752110 14387 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:20.752163 14387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:41:20.752354 14387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:41:20.754182 14387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/data/instance:
uuid: "48fe5a20c28142bca8ee7a378df77b86"
format_stamp: "Formatted at 2026-09-27 16:41:20 on dist-test-slave-z40b"
server_key: "d61f39e79df0a36ac517dc486f7baa7d"
server_key_iv: "aaef5b44e3fa8535c4f6bf7030388f7d"
server_key_version: "encryptionkey@0"
I20260927 16:41:20.754623 14387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/wal/instance:
uuid: "48fe5a20c28142bca8ee7a378df77b86"
format_stamp: "Formatted at 2026-09-27 16:41:20 on dist-test-slave-z40b"
server_key: "d61f39e79df0a36ac517dc486f7baa7d"
server_key_iv: "aaef5b44e3fa8535c4f6bf7030388f7d"
server_key_version: "encryptionkey@0"
I20260927 16:41:20.757783 14387 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:41:20.759783 14402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:20.760839 14387 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:41:20.760949 14387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "48fe5a20c28142bca8ee7a378df77b86"
format_stamp: "Formatted at 2026-09-27 16:41:20 on dist-test-slave-z40b"
server_key: "d61f39e79df0a36ac517dc486f7baa7d"
server_key_iv: "aaef5b44e3fa8535c4f6bf7030388f7d"
server_key_version: "encryptionkey@0"
I20260927 16:41:20.761044 14387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:20.770331 14387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:20.772878 14387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:20.773051 14387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:20.773553 14387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:41:20.774226 14387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:41:20.774272 14387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:20.774330 14387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:41:20.774363 14387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:20.782279 14387 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36877
I20260927 16:41:20.782301 14515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36877 every 8 connection(s)
I20260927 16:41:20.783037 14387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:41:20.786629 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 14387
I20260927 16:41:20.786760 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:41:20.787062 392 external_mini_cluster.cc:1428] Setting key fc3513cdb7da8940ef3df66245518057
Sep 27 16:41:20 dist-test-slave-z40b krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 16:41:20.795730 14230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:20.784652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58937 (local address 127.0.98.62:45119)
0927 16:41:20.784770 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:20.784774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:20.785425 (+ 651us) server_negotiation.cc:407] Connection header received
0927 16:41:20.786585 (+ 1160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:20.786589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:20.786645 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:20.786727 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:20.788027 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:20.788688 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:20.789745 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:20.789960 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:20.790067 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:20.792363 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:20.792373 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:20.792377 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:20.792399 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:20.793763 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:20.794239 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:20.794244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:20.794246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:20.794323 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:20.794638 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:20.794643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:20.794645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:20.794819 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:20.794919 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:20.795450 (+ 531us) server_negotiation.cc:299] Negotiation successful
0927 16:41:20.795565 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 16:41:20.796461 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:20.784907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:58937)
0927 16:41:20.785295 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:20.785331 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:20.786391 (+ 1060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:20.786930 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:20.786943 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:20.787366 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:41:20.787880 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:20.787893 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:20.788800 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:20.788804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:20.789618 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:20.789627 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:20.790095 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:20.790099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:20.790266 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:20.790763 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:41:20.790778 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:41:20.792215 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:41:20.793875 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:20.793880 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:20.793892 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:20.794127 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:20.794411 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:20.794415 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:20.794417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:20.794547 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:20.794930 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:41:20.794937 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:41:20.795145 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 16:41:20.795393 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 16:41:20.795689 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":103,"threads_started":1}
I20260927 16:41:20.797439 14516 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45119
I20260927 16:41:20.797659 14516 heartbeater.cc:464] Registering TS with master...
I20260927 16:41:20.798138 14516 heartbeater.cc:511] Master 127.0.98.62:45119 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 16:41:20.799094 14191 ts_manager.cc:194] Registered new tserver with Master: 48fe5a20c28142bca8ee7a378df77b86 (127.0.98.2:36877)
I20260927 16:41:20.799906 14191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:41:20.813462 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45119
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:41:20.917640 14521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:20.917865 14521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:20.917903 14521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:20.921588 14521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:20.921629 14521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:20.921702 14521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:41:20.926447 14521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:20.927312 14521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:20.928206 14521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:41:20.937036 14527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:20.937052 14526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:20.937055 14529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:20.937117 14521 server_base.cc:1034] running on GCE node
I20260927 16:41:20.937575 14521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:20.938163 14521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:20.939335 14521 hybrid_clock.cc:648] HybridClock initialized: now 1790527280939315 us; error 30 us; skew 500 ppm
Sep 27 16:41:20 dist-test-slave-z40b krb5kdc[14146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:20.941821 14521 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:41:20.942729 14521 webserver.cc:460] Webserver started at http://127.0.98.3:42177/ using document root <none> and password file <none>
I20260927 16:41:20.943282 14521 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:20.943377 14521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:41:20.943581 14521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:41:20.945353 14521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/data/instance:
uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8"
format_stamp: "Formatted at 2026-09-27 16:41:20 on dist-test-slave-z40b"
server_key: "620a1470d0fce7be2f68645d1eed3ab6"
server_key_iv: "fde9ec72ddbf9c6cad9fd2d7df8e3394"
server_key_version: "encryptionkey@0"
I20260927 16:41:20.945892 14521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8"
format_stamp: "Formatted at 2026-09-27 16:41:20 on dist-test-slave-z40b"
server_key: "620a1470d0fce7be2f68645d1eed3ab6"
server_key_iv: "fde9ec72ddbf9c6cad9fd2d7df8e3394"
server_key_version: "encryptionkey@0"
I20260927 16:41:20.949215 14521 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:41:20.951299 14536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:20.952415 14521 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:41:20.952549 14521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8"
format_stamp: "Formatted at 2026-09-27 16:41:20 on dist-test-slave-z40b"
server_key: "620a1470d0fce7be2f68645d1eed3ab6"
server_key_iv: "fde9ec72ddbf9c6cad9fd2d7df8e3394"
server_key_version: "encryptionkey@0"
I20260927 16:41:20.952644 14521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:20.985293 14521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:20.987898 14521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:20.988090 14521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:20.988636 14521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:41:20.989392 14521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:41:20.989463 14521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:20.989527 14521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:41:20.989576 14521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:20.997745 14521 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39537
I20260927 16:41:20.997788 14649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39537 every 8 connection(s)
I20260927 16:41:20.998574 14521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:41:20.998884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 14521
I20260927 16:41:20.999001 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:41:20.999310 392 external_mini_cluster.cc:1428] Setting key 48203e5afad6cd9405424e7734c7109c
Sep 27 16:41:21 dist-test-slave-z40b krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:41:21.011045 14230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.000205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58131 (local address 127.0.98.62:45119)
0927 16:41:21.000331 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:21.000335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:21.001211 (+ 876us) server_negotiation.cc:407] Connection header received
0927 16:41:21.002588 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:21.002591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:21.002644 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:21.002719 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:21.003857 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.004376 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.005476 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.005704 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.005809 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.008123 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:21.008127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:21.008129 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:21.008151 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:21.009391 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:21.009820 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:21.009822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:21.009823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:21.009891 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:21.010175 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:21.010177 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:21.010178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:21.010353 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:21.010423 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:21.010840 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 16:41:21.010929 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:41:21.011880 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.000595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:58131)
0927 16:41:21.001075 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:21.001117 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:21.002394 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:21.002880 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:21.002886 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:21.003259 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:41:21.003709 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.003720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.004503 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.004506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.005352 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.005360 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.005840 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.005844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.006024 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.006547 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:41:21.006562 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:41:21.007977 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:41:21.009488 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:21.009493 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:21.009502 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:21.009738 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:21.009995 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:21.009997 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:21.009998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:21.010107 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:21.010432 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:41:21.010436 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:41:21.010632 (+ 196us) client_negotiation.cc:769] Sending connection context
0927 16:41:21.010794 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 16:41:21.011009 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":134,"threads_started":1}
I20260927 16:41:21.013001 14650 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45119
I20260927 16:41:21.013202 14650 heartbeater.cc:464] Registering TS with master...
I20260927 16:41:21.013605 14650 heartbeater.cc:511] Master 127.0.98.62:45119 requested a full tablet report, sending...
I20260927 16:41:21.014501 14191 ts_manager.cc:194] Registered new tserver with Master: 3b5a90dc9f9d41de9b4fabb7c5e21be8 (127.0.98.3:39537)
I20260927 16:41:21.015044 14191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58131
I20260927 16:41:21.024644 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:41:21.033609 14230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.026413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.0.98.62:45119)
0927 16:41:21.026552 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:21.026557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:21.026745 (+ 188us) server_negotiation.cc:407] Connection header received
0927 16:41:21.026884 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:21.026887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:21.026928 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:21.027018 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:21.027857 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.028370 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.029287 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.029507 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.029628 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.030840 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:21.030844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:21.030847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:21.030870 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:21.032033 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:21.032435 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:21.032439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:21.032440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:21.032516 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:21.032779 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:21.032783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:21.032785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:21.032984 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:21.033124 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:21.033346 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 16:41:21.033453 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:41:21.036104 14191 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:41:21.038888 14191 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:41:21.052166 14661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.046525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33767 (local address 127.0.0.1:45166)
0927 16:41:21.046840 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:21.046889 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:21.047003 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:21.048127 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:21.048135 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:21.048156 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:41:21.048299 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.048306 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.049370 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.049376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.050508 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.050518 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.051657 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.051661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.051772 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.051807 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 16:41:21.051932 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 16:41:21.052031 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":85,"thread_start_us":76,"threads_started":1}
I20260927 16:41:21.052188 14662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.046715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45166 (local address 127.0.98.1:33767)
0927 16:41:21.047622 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:21.047633 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:21.047656 (+ 23us) server_negotiation.cc:407] Connection header received
0927 16:41:21.047749 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:21.047758 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:21.047956 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:21.048461 (+ 505us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:41:21.048590 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.049252 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.050648 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.051542 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.051644 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.051785 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:21.051972 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 16:41:21.052058 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":811,"thread_start_us":84,"threads_started":1}
I20260927 16:41:21.052774 14664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.047253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39537 (local address 127.0.0.1:57586)
0927 16:41:21.047640 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:21.047653 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:21.047734 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:21.048370 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:21.048373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:21.048401 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:41:21.048499 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.048505 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.050096 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.050102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.051362 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.051376 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.052416 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.052421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.052519 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.052542 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 16:41:21.052593 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 16:41:21.052654 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":3,"spinlock_wait_cycles":4736,"thread_start_us":82,"threads_started":1}
I20260927 16:41:21.052922 14663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.046901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36877 (local address 127.0.0.1:49432)
0927 16:41:21.047294 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:21.047307 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:21.047391 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:21.048370 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:21.048373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:21.048403 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:41:21.048642 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.048647 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.050015 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.050018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.051287 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.051297 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.052547 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.052550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.052608 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.052623 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 16:41:21.052707 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 16:41:21.052772 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":235,"spinlock_wait_cycles":4608,"thread_start_us":86,"threads_started":1}
I20260927 16:41:21.052995 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.047227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49432 (local address 127.0.98.2:36877)
0927 16:41:21.047921 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:21.047932 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:21.047953 (+ 21us) server_negotiation.cc:407] Connection header received
0927 16:41:21.048027 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:21.048032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:21.048201 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:21.048535 (+ 334us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:41:21.048756 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.049873 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.051422 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.052277 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.052379 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.052500 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:21.052736 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 16:41:21.052825 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":134,"threads_started":1}
I20260927 16:41:21.053438 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.047396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57586 (local address 127.0.98.3:39537)
0927 16:41:21.047958 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:21.047969 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:21.047988 (+ 19us) server_negotiation.cc:407] Connection header received
0927 16:41:21.048043 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:21.048047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:21.048210 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:21.048804 (+ 594us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:41:21.048943 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.049967 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.051496 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.052292 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.053024 (+ 732us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.053141 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:21.053186 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 16:41:21.053283 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":121,"threads_started":1}
I20260927 16:41:21.055215 14316 tablet_service.cc:1467] Processing CreateTablet for tablet d9ccf7a11ab84cba9c4aa75eebfb451a (DEFAULT_TABLE table=test-table [id=932c7b7501544ab68b589e54e6b38db7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:41:21.055570 14450 tablet_service.cc:1467] Processing CreateTablet for tablet d9ccf7a11ab84cba9c4aa75eebfb451a (DEFAULT_TABLE table=test-table [id=932c7b7501544ab68b589e54e6b38db7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:41:21.055709 14584 tablet_service.cc:1467] Processing CreateTablet for tablet d9ccf7a11ab84cba9c4aa75eebfb451a (DEFAULT_TABLE table=test-table [id=932c7b7501544ab68b589e54e6b38db7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:41:21.056401 14316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9ccf7a11ab84cba9c4aa75eebfb451a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:41:21.056571 14450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9ccf7a11ab84cba9c4aa75eebfb451a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:41:21.056576 14584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9ccf7a11ab84cba9c4aa75eebfb451a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:41:21.062748 14667 tablet_bootstrap.cc:492] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105: Bootstrap starting.
I20260927 16:41:21.063117 14668 tablet_bootstrap.cc:492] T d9ccf7a11ab84cba9c4aa75eebfb451a P 3b5a90dc9f9d41de9b4fabb7c5e21be8: Bootstrap starting.
I20260927 16:41:21.063978 14669 tablet_bootstrap.cc:492] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86: Bootstrap starting.
I20260927 16:41:21.065194 14668 tablet_bootstrap.cc:654] T d9ccf7a11ab84cba9c4aa75eebfb451a P 3b5a90dc9f9d41de9b4fabb7c5e21be8: Neither blocks nor log segments found. Creating new log.
I20260927 16:41:21.065220 14667 tablet_bootstrap.cc:654] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105: Neither blocks nor log segments found. Creating new log.
I20260927 16:41:21.066028 14669 tablet_bootstrap.cc:654] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86: Neither blocks nor log segments found. Creating new log.
I20260927 16:41:21.066445 14667 log.cc:824] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105: Log is configured to *not* fsync() on all Append() calls
I20260927 16:41:21.066444 14668 log.cc:824] T d9ccf7a11ab84cba9c4aa75eebfb451a P 3b5a90dc9f9d41de9b4fabb7c5e21be8: Log is configured to *not* fsync() on all Append() calls
I20260927 16:41:21.066735 14669 log.cc:824] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86: Log is configured to *not* fsync() on all Append() calls
I20260927 16:41:21.068372 14667 tablet_bootstrap.cc:492] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105: No bootstrap required, opened a new log
I20260927 16:41:21.068384 14668 tablet_bootstrap.cc:492] T d9ccf7a11ab84cba9c4aa75eebfb451a P 3b5a90dc9f9d41de9b4fabb7c5e21be8: No bootstrap required, opened a new log
I20260927 16:41:21.068572 14667 ts_tablet_manager.cc:1397] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:41:21.068583 14668 ts_tablet_manager.cc:1397] T d9ccf7a11ab84cba9c4aa75eebfb451a P 3b5a90dc9f9d41de9b4fabb7c5e21be8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:41:21.071169 14667 raft_consensus.cc:348] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } } peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260927 16:41:21.071169 14668 raft_consensus.cc:348] T d9ccf7a11ab84cba9c4aa75eebfb451a P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } } peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260927 16:41:21.071465 14668 raft_consensus.cc:374] T d9ccf7a11ab84cba9c4aa75eebfb451a P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:41:21.071465 14667 raft_consensus.cc:374] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:41:21.071527 14667 raft_consensus.cc:729] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbff7cfa13fc4c1ab07b892db9b7f105, State: Initialized, Role: FOLLOWER
I20260927 16:41:21.071528 14668 raft_consensus.cc:729] T d9ccf7a11ab84cba9c4aa75eebfb451a P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5a90dc9f9d41de9b4fabb7c5e21be8, State: Initialized, Role: FOLLOWER
I20260927 16:41:21.071950 14667 consensus_queue.cc:260] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } } peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260927 16:41:21.071976 14668 consensus_queue.cc:260] T d9ccf7a11ab84cba9c4aa75eebfb451a P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } } peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260927 16:41:21.072589 14382 heartbeater.cc:503] Master 127.0.98.62:45119 was elected leader, sending a full tablet report...
I20260927 16:41:21.072687 14669 tablet_bootstrap.cc:492] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86: No bootstrap required, opened a new log
I20260927 16:41:21.072837 14669 ts_tablet_manager.cc:1397] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260927 16:41:21.072898 14668 ts_tablet_manager.cc:1428] T d9ccf7a11ab84cba9c4aa75eebfb451a P 3b5a90dc9f9d41de9b4fabb7c5e21be8: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 16:41:21.072901 14667 ts_tablet_manager.cc:1428] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 16:41:21.073175 14650 heartbeater.cc:503] Master 127.0.98.62:45119 was elected leader, sending a full tablet report...
I20260927 16:41:21.076552 14669 raft_consensus.cc:348] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } } peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260927 16:41:21.076830 14669 raft_consensus.cc:374] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:41:21.076895 14669 raft_consensus.cc:729] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48fe5a20c28142bca8ee7a378df77b86, State: Initialized, Role: FOLLOWER
I20260927 16:41:21.077306 14669 consensus_queue.cc:260] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } } peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260927 16:41:21.077950 14516 heartbeater.cc:503] Master 127.0.98.62:45119 was elected leader, sending a full tablet report...
I20260927 16:41:21.078166 14669 ts_tablet_manager.cc:1428] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260927 16:41:21.086443 14383 tablet.cc:2369] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:41:21.093566 14675 raft_consensus.cc:482] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:41:21.093729 14675 raft_consensus.cc:504] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } } peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260927 16:41:21.094799 14675 leader_election.cc:288] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbff7cfa13fc4c1ab07b892db9b7f105 (127.0.98.1:33767), 3b5a90dc9f9d41de9b4fabb7c5e21be8 (127.0.98.3:39537)
I20260927 16:41:21.097918 14662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.095052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35125 (local address 127.0.98.1:33767)
0927 16:41:21.095175 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:21.095179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:21.095225 (+ 46us) server_negotiation.cc:407] Connection header received
0927 16:41:21.095277 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:21.095281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:21.095328 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:21.095407 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:41:21.095679 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.096129 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.097056 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.097418 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.097527 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.097580 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:21.097751 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 16:41:21.097815 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:41:21.097918 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.094965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33767 (local address 127.0.98.2:35125)
0927 16:41:21.095060 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:21.095074 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:21.095154 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:21.095423 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:21.095426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:21.095443 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:41:21.095558 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.095578 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.096239 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.096243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.096937 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.096945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.097528 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.097531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.097625 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.097645 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 16:41:21.097733 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 16:41:21.097802 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 16:41:21.098376 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.095234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39537 (local address 127.0.98.2:51507)
0927 16:41:21.095607 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:21.095616 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:21.095723 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:21.095926 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:21.095928 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:21.095942 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:41:21.096054 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.096058 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.096682 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.096694 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.097382 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.097390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.097995 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.097998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.098109 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.098131 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 16:41:21.098202 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 16:41:21.098257 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":127,"threads_started":1}
I20260927 16:41:21.098394 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.095305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51507 (local address 127.0.98.3:39537)
0927 16:41:21.095440 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:21.095445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:21.095690 (+ 245us) server_negotiation.cc:407] Connection header received
0927 16:41:21.095798 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:21.095801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:21.095842 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:21.095918 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:41:21.096151 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.096586 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.097495 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.097868 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.097982 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.098034 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:21.098217 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 16:41:21.098293 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 16:41:21.098471 14336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ccf7a11ab84cba9c4aa75eebfb451a" candidate_uuid: "48fe5a20c28142bca8ee7a378df77b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" is_pre_election: true
I20260927 16:41:21.098748 14336 raft_consensus.cc:2454] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48fe5a20c28142bca8ee7a378df77b86 in term 0.
I20260927 16:41:21.098874 14604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ccf7a11ab84cba9c4aa75eebfb451a" candidate_uuid: "48fe5a20c28142bca8ee7a378df77b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" is_pre_election: true
I20260927 16:41:21.099141 14604 raft_consensus.cc:2454] T d9ccf7a11ab84cba9c4aa75eebfb451a P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48fe5a20c28142bca8ee7a378df77b86 in term 0.
I20260927 16:41:21.099239 14406 leader_election.cc:302] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48fe5a20c28142bca8ee7a378df77b86, bbff7cfa13fc4c1ab07b892db9b7f105; no voters:
I20260927 16:41:21.099493 14675 raft_consensus.cc:2781] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:41:21.099557 14675 raft_consensus.cc:482] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:41:21.099768 14675 raft_consensus.cc:3037] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:41:21.100742 14675 raft_consensus.cc:504] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } } peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260927 16:41:21.101037 14675 leader_election.cc:288] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86 [CANDIDATE]: Term 1 election: Requested vote from peers bbff7cfa13fc4c1ab07b892db9b7f105 (127.0.98.1:33767), 3b5a90dc9f9d41de9b4fabb7c5e21be8 (127.0.98.3:39537)
I20260927 16:41:21.101365 14336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ccf7a11ab84cba9c4aa75eebfb451a" candidate_uuid: "48fe5a20c28142bca8ee7a378df77b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105"
I20260927 16:41:21.101450 14604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ccf7a11ab84cba9c4aa75eebfb451a" candidate_uuid: "48fe5a20c28142bca8ee7a378df77b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8"
I20260927 16:41:21.101491 14336 raft_consensus.cc:3037] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:41:21.101579 14604 raft_consensus.cc:3037] T d9ccf7a11ab84cba9c4aa75eebfb451a P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:41:21.102476 14336 raft_consensus.cc:2454] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48fe5a20c28142bca8ee7a378df77b86 in term 1.
I20260927 16:41:21.102550 14604 raft_consensus.cc:2454] T d9ccf7a11ab84cba9c4aa75eebfb451a P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48fe5a20c28142bca8ee7a378df77b86 in term 1.
I20260927 16:41:21.102751 14406 leader_election.cc:302] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48fe5a20c28142bca8ee7a378df77b86, bbff7cfa13fc4c1ab07b892db9b7f105; no voters:
I20260927 16:41:21.102905 14675 raft_consensus.cc:2781] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:41:21.103107 14675 raft_consensus.cc:686] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86 [term 1 LEADER]: Becoming Leader. State: Replica: 48fe5a20c28142bca8ee7a378df77b86, State: Running, Role: LEADER
I20260927 16:41:21.103417 14675 consensus_queue.cc:237] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } } peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260927 16:41:21.105942 14191 catalog_manager.cc:5495] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48fe5a20c28142bca8ee7a378df77b86 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "48fe5a20c28142bca8ee7a378df77b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } health_report { overall_health: HEALTHY } } }
I20260927 16:41:21.129420 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.125796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49438 (local address 127.0.98.2:36877)
0927 16:41:21.125925 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:21.125930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:21.125944 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:41:21.126134 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:21.126136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:21.126186 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:21.126294 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:41:21.126532 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.127008 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.127975 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.128209 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.128325 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.128655 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:41:21.129054 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:41:21.129175 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:21.129245 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 16:41:21.129295 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:41:21.134728 14604 raft_consensus.cc:1261] T d9ccf7a11ab84cba9c4aa75eebfb451a P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [term 1 FOLLOWER]: Refusing update from remote peer 48fe5a20c28142bca8ee7a378df77b86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:41:21.134826 14336 raft_consensus.cc:1261] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105 [term 1 FOLLOWER]: Refusing update from remote peer 48fe5a20c28142bca8ee7a378df77b86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:41:21.135403 14675 consensus_queue.cc:1035] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:41:21.135555 14677 consensus_queue.cc:1035] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:41:21.143872 14680 mvcc.cc:205] Tried to move back new op lower bound from 7333999743521161216 to 7333999743401742336. Current Snapshot: MvccSnapshot[applied={T|T < 7333999743521161216}]
I20260927 16:41:21.145810 14681 mvcc.cc:205] Tried to move back new op lower bound from 7333999743521161216 to 7333999743401742336. Current Snapshot: MvccSnapshot[applied={T|T < 7333999743521161216 or (T in {7333999743521161216})}]
I20260927 16:41:21.152655 14191 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:41:21.152832 14191 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:41:21.155167 14191 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58: Sending DeleteTablet for 3 replicas of tablet d9ccf7a11ab84cba9c4aa75eebfb451a
I20260927 16:41:21.155900 14450 tablet_service.cc:1514] Processing DeleteTablet for tablet d9ccf7a11ab84cba9c4aa75eebfb451a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:41:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49432
I20260927 16:41:21.155911 14584 tablet_service.cc:1514] Processing DeleteTablet for tablet d9ccf7a11ab84cba9c4aa75eebfb451a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:41:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57586
I20260927 16:41:21.155995 14316 tablet_service.cc:1514] Processing DeleteTablet for tablet d9ccf7a11ab84cba9c4aa75eebfb451a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:41:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45166
I20260927 16:41:21.156358 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 14253
I20260927 16:41:21.156380 14690 tablet_replica.cc:330] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86: stopping tablet replica
I20260927 16:41:21.156713 14690 raft_consensus.cc:2229] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:41:21.157112 14690 raft_consensus.cc:2258] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:41:21.158325 14690 ts_tablet_manager.cc:1905] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:41:21.160493 14690 ts_tablet_manager.cc:1918] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:41:21.160595 14690 log.cc:1196] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/wal/wals/d9ccf7a11ab84cba9c4aa75eebfb451a
I20260927 16:41:21.160848 14691 tablet_replica.cc:330] T d9ccf7a11ab84cba9c4aa75eebfb451a P 3b5a90dc9f9d41de9b4fabb7c5e21be8: stopping tablet replica
I20260927 16:41:21.160971 14690 ts_tablet_manager.cc:1939] T d9ccf7a11ab84cba9c4aa75eebfb451a P 48fe5a20c28142bca8ee7a378df77b86: Deleting consensus metadata
I20260927 16:41:21.161269 14691 raft_consensus.cc:2229] T d9ccf7a11ab84cba9c4aa75eebfb451a P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:41:21.161595 14691 raft_consensus.cc:2258] T d9ccf7a11ab84cba9c4aa75eebfb451a P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:41:21.162061 14176 catalog_manager.cc:4841] TS 48fe5a20c28142bca8ee7a378df77b86 (127.0.98.2:36877): tablet d9ccf7a11ab84cba9c4aa75eebfb451a (table test-table [id=932c7b7501544ab68b589e54e6b38db7]) successfully deleted
I20260927 16:41:21.162928 14691 ts_tablet_manager.cc:1905] T d9ccf7a11ab84cba9c4aa75eebfb451a P 3b5a90dc9f9d41de9b4fabb7c5e21be8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:41:21.165233 14691 ts_tablet_manager.cc:1918] T d9ccf7a11ab84cba9c4aa75eebfb451a P 3b5a90dc9f9d41de9b4fabb7c5e21be8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:41:21.165328 14691 log.cc:1196] T d9ccf7a11ab84cba9c4aa75eebfb451a P 3b5a90dc9f9d41de9b4fabb7c5e21be8: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/wal/wals/d9ccf7a11ab84cba9c4aa75eebfb451a
W20260927 16:41:21.165441 14178 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33767 (error 108)
I20260927 16:41:21.165664 14691 ts_tablet_manager.cc:1939] T d9ccf7a11ab84cba9c4aa75eebfb451a P 3b5a90dc9f9d41de9b4fabb7c5e21be8: Deleting consensus metadata
I20260927 16:41:21.166412 14663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.166142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33767 (local address 127.0.0.1:45172)
0927 16:41:21.166272 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:21.166338 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 16:41:21.166406 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 14387
W20260927 16:41:21.167861 14178 catalog_manager.cc:4584] TS bbff7cfa13fc4c1ab07b892db9b7f105 (127.0.98.1:33767): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d9ccf7a11ab84cba9c4aa75eebfb451a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33767: connect: Connection refused (error 111)
I20260927 16:41:21.168081 14178 catalog_manager.cc:4841] TS 3b5a90dc9f9d41de9b4fabb7c5e21be8 (127.0.98.3:39537): tablet d9ccf7a11ab84cba9c4aa75eebfb451a (table test-table [id=932c7b7501544ab68b589e54e6b38db7]) successfully deleted
I20260927 16:41:21.172114 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 14521
I20260927 16:41:21.177078 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 14159
I20260927 16:41:21.183043 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45119
--webserver_interface=127.0.98.62
--webserver_port=39371
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:41:21.287811 14694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:21.288115 14694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:21.288172 14694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:21.292218 14694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:41:21.292268 14694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:21.292287 14694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:21.292302 14694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:41:21.292317 14694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:41:21.297256 14694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:21.298069 14694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:21.298975 14694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:41:21.311160 14700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:21.311285 14694 server_base.cc:1034] running on GCE node
W20260927 16:41:21.311231 14701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:21.311230 14703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:21.312043 14694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:21.313011 14694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:21.314194 14694 hybrid_clock.cc:648] HybridClock initialized: now 1790527281314175 us; error 26 us; skew 500 ppm
Sep 27 16:41:21 dist-test-slave-z40b krb5kdc[14146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:21.316784 14694 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:41:21.317543 14694 webserver.cc:460] Webserver started at http://127.0.98.62:39371/ using document root <none> and password file <none>
I20260927 16:41:21.318012 14694 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:21.318071 14694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:41:21.321589 14694 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:41:21.323557 14710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:21.324751 14694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:41:21.324877 14694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "ed559add211b4fe190913711a83ace58"
format_stamp: "Formatted at 2026-09-27 16:41:20 on dist-test-slave-z40b"
server_key: "7dcc356e390aaab0e210d29dd3a38588"
server_key_iv: "543efba9c1852c62580d11adf0a57e14"
server_key_version: "encryptionkey@0"
I20260927 16:41:21.325443 14694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:21.363483 14694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:21.366148 14694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:21.366353 14694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:21.373284 14694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45119
I20260927 16:41:21.373550 14762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45119 every 8 connection(s)
I20260927 16:41:21.374399 14694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:41:21.378463 14763 sys_catalog.cc:263] Verifying existing consensus state
I20260927 16:41:21.379331 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 14694
I20260927 16:41:21.379820 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33767
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45119
--webserver_port=33135
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:41:21.380446 14763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58: Bootstrap starting.
I20260927 16:41:21.396836 14763 log.cc:824] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58: Log is configured to *not* fsync() on all Append() calls
I20260927 16:41:21.405050 14763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:41:21.405403 14763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58: Bootstrap complete.
I20260927 16:41:21.409219 14763 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed559add211b4fe190913711a83ace58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } }
I20260927 16:41:21.409637 14763 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed559add211b4fe190913711a83ace58, State: Initialized, Role: FOLLOWER
I20260927 16:41:21.410087 14763 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed559add211b4fe190913711a83ace58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } }
I20260927 16:41:21.410233 14763 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:41:21.410317 14763 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:41:21.410434 14763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [term 1 FOLLOWER]: Advancing to term 2
I20260927 16:41:21.412302 14763 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed559add211b4fe190913711a83ace58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } }
I20260927 16:41:21.412606 14763 leader_election.cc:302] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed559add211b4fe190913711a83ace58; no voters:
I20260927 16:41:21.412902 14763 leader_election.cc:288] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 16:41:21.412976 14767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 16:41:21.413156 14767 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [term 2 LEADER]: Becoming Leader. State: Replica: ed559add211b4fe190913711a83ace58, State: Running, Role: LEADER
I20260927 16:41:21.413475 14767 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed559add211b4fe190913711a83ace58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } }
I20260927 16:41:21.414019 14763 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:41:21.414388 14767 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed559add211b4fe190913711a83ace58. Latest consensus state: current_term: 2 leader_uuid: "ed559add211b4fe190913711a83ace58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed559add211b4fe190913711a83ace58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } } }
I20260927 16:41:21.414491 14767 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [sys.catalog]: This master's current role is: LEADER
I20260927 16:41:21.414834 14768 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ed559add211b4fe190913711a83ace58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed559add211b4fe190913711a83ace58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } } }
I20260927 16:41:21.414981 14768 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [sys.catalog]: This master's current role is: LEADER
I20260927 16:41:21.415403 14771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:41:21.419888 14771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:41:21.420467 14771 catalog_manager.cc:1259] Loaded cluster ID: 06701aaea5784b5385b3207bbd6a9839
I20260927 16:41:21.420549 14771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:41:21.421954 14771 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 16:41:21.426812 14771 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 16:41:21.441519 14771 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:41:21.442225 14771 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58: Loaded TSK: 0
I20260927 16:41:21.443338 14771 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 16:41:21.487582 14765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:21.487784 14765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:21.487823 14765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:21.491706 14765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:21.491750 14765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:21.491832 14765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:41:21.496809 14765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33767
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33135
--tserver_master_addrs=127.0.98.62:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:21.497649 14765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:21.498548 14765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:41:21.507020 14790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:21.507017 14791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:21.507020 14793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:21.507926 14765 server_base.cc:1034] running on GCE node
I20260927 16:41:21.508365 14765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:21.509032 14765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:21.510214 14765 hybrid_clock.cc:648] HybridClock initialized: now 1790527281510185 us; error 46 us; skew 500 ppm
Sep 27 16:41:21 dist-test-slave-z40b krb5kdc[14146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:21.513340 14765 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:41:21.514156 14765 webserver.cc:460] Webserver started at http://127.0.98.1:33135/ using document root <none> and password file <none>
I20260927 16:41:21.514711 14765 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:21.514780 14765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:41:21.518476 14765 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:41:21.520591 14800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:21.521787 14765 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:41:21.521917 14765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "bbff7cfa13fc4c1ab07b892db9b7f105"
format_stamp: "Formatted at 2026-09-27 16:41:20 on dist-test-slave-z40b"
server_key: "e017c70c57309ce7d4c9835237eee7e6"
server_key_iv: "abe5007fb5b487bc4768842fe1a91804"
server_key_version: "encryptionkey@0"
I20260927 16:41:21.522446 14765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:21.532956 14765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:21.535778 14765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:21.535948 14765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:21.536432 14765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:41:21.537431 14807 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 16:41:21.541082 14765 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 16:41:21.541143 14765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260927 16:41:21.541184 14765 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 16:41:21.542670 14765 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 16:41:21.542719 14765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 16:41:21.542943 14807 tablet_bootstrap.cc:492] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105: Bootstrap starting.
I20260927 16:41:21.552820 14765 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33767
I20260927 16:41:21.552949 14914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33767 every 8 connection(s)
I20260927 16:41:21.553953 14765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:41:21.555709 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 14765
I20260927 16:41:21.556116 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36877
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45119
--webserver_port=39323
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:41:21.563321 14807 log.cc:824] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105: Log is configured to *not* fsync() on all Append() calls
Sep 27 16:41:21 dist-test-slave-z40b krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:41:21.571750 14807 tablet_bootstrap.cc:492] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:41:21.572149 14807 tablet_bootstrap.cc:492] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105: Bootstrap complete.
I20260927 16:41:21.572991 14917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.556030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:55899)
0927 16:41:21.559184 (+ 3154us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:21.559281 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:21.560164 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:21.561765 (+ 1601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:21.561773 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:21.562277 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:41:21.562793 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.562806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.564335 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.564338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.565100 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.565108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.565849 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.565853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.566028 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.566606 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:41:21.566619 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:41:21.568051 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:41:21.569986 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:21.569991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:21.570002 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:21.570264 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:21.570604 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:21.570606 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:21.570608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:21.570759 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:21.571380 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:41:21.571386 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:41:21.571619 (+ 233us) client_negotiation.cc:769] Sending connection context
0927 16:41:21.571792 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 16:41:21.572173 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2958,"thread_start_us":142,"threads_started":1}
I20260927 16:41:21.572989 14919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.557239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55899 (local address 127.0.98.62:45119)
0927 16:41:21.557644 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:21.557661 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:21.559425 (+ 1764us) server_negotiation.cc:407] Connection header received
0927 16:41:21.560651 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:21.560664 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:21.561020 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:21.561344 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:21.562973 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.564161 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.565234 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.565654 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.565883 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.568230 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:21.568236 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:21.568256 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:21.568281 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:21.569857 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:21.570378 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:21.570384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:21.570390 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:21.570500 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:21.570853 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:21.570858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:21.570860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:21.571171 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:21.571407 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:21.571939 (+ 532us) server_negotiation.cc:299] Negotiation successful
0927 16:41:21.572173 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":98,"threads_started":1}
I20260927 16:41:21.573503 14807 ts_tablet_manager.cc:1397] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105: Time spent bootstrapping tablet: real 0.031s user 0.017s sys 0.011s
I20260927 16:41:21.575490 14915 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45119
I20260927 16:41:21.575695 14915 heartbeater.cc:464] Registering TS with master...
I20260927 16:41:21.576107 14915 heartbeater.cc:511] Master 127.0.98.62:45119 requested a full tablet report, sending...
I20260927 16:41:21.576256 14807 raft_consensus.cc:348] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } } peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260927 16:41:21.576756 14807 raft_consensus.cc:729] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbff7cfa13fc4c1ab07b892db9b7f105, State: Initialized, Role: FOLLOWER
I20260927 16:41:21.577199 14807 consensus_queue.cc:260] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } } peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260927 16:41:21.578178 14807 ts_tablet_manager.cc:1428] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:41:21.579757 14727 ts_manager.cc:194] Registered new tserver with Master: bbff7cfa13fc4c1ab07b892db9b7f105 (127.0.98.1:33767)
I20260927 16:41:21.582810 14727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55899
I20260927 16:41:21.584955 14915 heartbeater.cc:503] Master 127.0.98.62:45119 was elected leader, sending a full tablet report...
Sep 27 16:41:21 dist-test-slave-z40b krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 16:41:21.592334 14922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.582332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33767 (local address 127.0.0.1:45180)
0927 16:41:21.582959 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:21.582994 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:21.583108 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:21.583759 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:21.583765 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:21.584132 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:41:21.584242 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.584250 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.585413 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.585417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.586310 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.586321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.586821 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.586835 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.586935 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.587485 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:41:21.587500 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:41:21.588876 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:41:21.590810 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:21.590817 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:21.590820 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:21.591026 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:21.591348 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:21.591351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:21.591352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:21.591426 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:21.591887 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:41:21.591891 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:41:21.591991 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 16:41:21.592094 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 16:41:21.592213 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":108,"threads_started":1}
I20260927 16:41:21.592682 14923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.582424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45180 (local address 127.0.98.1:33767)
0927 16:41:21.582720 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:21.582750 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:21.583083 (+ 333us) server_negotiation.cc:407] Connection header received
0927 16:41:21.583284 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:21.583289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:21.583464 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:21.583589 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:21.584374 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.585301 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.586425 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.586722 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.586802 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.589005 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:21.589013 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:21.589024 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:21.589048 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:21.590680 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:21.591128 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:21.591134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:21.591135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:21.591257 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:21.591518 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:21.591520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:21.591522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:21.591782 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:21.591902 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:21.592187 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 16:41:21.592329 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":88,"threads_started":1}
I20260927 16:41:21.593539 14849 tablet_service.cc:1514] Processing DeleteTablet for tablet d9ccf7a11ab84cba9c4aa75eebfb451a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:41:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45180
I20260927 16:41:21.594290 14924 tablet_replica.cc:330] stopping tablet replica
I20260927 16:41:21.594609 14924 raft_consensus.cc:2229] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:41:21.594861 14924 raft_consensus.cc:2258] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:41:21.595816 14924 ts_tablet_manager.cc:1905] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:41:21.599857 14924 ts_tablet_manager.cc:1918] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:41:21.599951 14924 log.cc:1196] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/wal/wals/d9ccf7a11ab84cba9c4aa75eebfb451a
I20260927 16:41:21.600248 14924 ts_tablet_manager.cc:1939] T d9ccf7a11ab84cba9c4aa75eebfb451a P bbff7cfa13fc4c1ab07b892db9b7f105: Deleting consensus metadata
I20260927 16:41:21.601918 14714 catalog_manager.cc:4841] TS bbff7cfa13fc4c1ab07b892db9b7f105 (127.0.98.1:33767): tablet d9ccf7a11ab84cba9c4aa75eebfb451a (table test-table [id=932c7b7501544ab68b589e54e6b38db7]) successfully deleted
W20260927 16:41:21.672329 14918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:21.672539 14918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:21.672628 14918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:21.676443 14918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:21.676487 14918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:21.676578 14918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:41:21.681614 14918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39323
--tserver_master_addrs=127.0.98.62:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:21.682551 14918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:21.683482 14918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:41:21.692144 14932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:21.692139 14929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:21.692137 14930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:21.692365 14918 server_base.cc:1034] running on GCE node
I20260927 16:41:21.692737 14918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:21.693305 14918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:21.694461 14918 hybrid_clock.cc:648] HybridClock initialized: now 1790527281694421 us; error 50 us; skew 500 ppm
Sep 27 16:41:21 dist-test-slave-z40b krb5kdc[14146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:21.697072 14918 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:41:21.697877 14918 webserver.cc:460] Webserver started at http://127.0.98.2:39323/ using document root <none> and password file <none>
I20260927 16:41:21.698388 14918 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:21.698474 14918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:41:21.702167 14918 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:41:21.704146 14939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:21.705233 14918 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:41:21.705320 14918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "48fe5a20c28142bca8ee7a378df77b86"
format_stamp: "Formatted at 2026-09-27 16:41:20 on dist-test-slave-z40b"
server_key: "d61f39e79df0a36ac517dc486f7baa7d"
server_key_iv: "aaef5b44e3fa8535c4f6bf7030388f7d"
server_key_version: "encryptionkey@0"
I20260927 16:41:21.705777 14918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:21.716635 14918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:21.719134 14918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:21.719326 14918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:21.719803 14918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:41:21.720491 14918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:41:21.720535 14918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:21.720570 14918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:41:21.720592 14918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:21.729316 14918 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36877
I20260927 16:41:21.729396 15052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36877 every 8 connection(s)
I20260927 16:41:21.730324 14918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:41:21.733592 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 14918
I20260927 16:41:21.733945 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39537
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45119
--webserver_port=42177
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 16:41:21 dist-test-slave-z40b krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:41:21.742738 14919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.732022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36413 (local address 127.0.98.62:45119)
0927 16:41:21.732141 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:21.732145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:21.732838 (+ 693us) server_negotiation.cc:407] Connection header received
0927 16:41:21.734065 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:21.734068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:21.734129 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:21.734212 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:21.735328 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.735843 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.736881 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.737104 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.737211 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.739424 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:21.739428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:21.739430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:21.739463 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:21.740812 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:21.741296 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:21.741299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:21.741301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:21.741379 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:21.741712 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:21.741716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:21.741717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:21.741886 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:21.741987 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:21.742438 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 16:41:21.742559 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 16:41:21.743512 15055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.732312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:36413)
0927 16:41:21.732707 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:21.732744 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:21.733874 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:21.734395 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:21.734405 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:21.734730 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:41:21.735143 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.735155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.735971 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.735975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.736762 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.736771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.737238 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.737242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.737392 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.737908 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:41:21.737921 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:41:21.739280 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:41:21.740924 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:21.740929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:21.740941 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:21.741177 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:21.741475 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:21.741478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:21.741480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:21.741608 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:21.741986 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:41:21.741992 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:41:21.742215 (+ 223us) client_negotiation.cc:769] Sending connection context
0927 16:41:21.742385 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 16:41:21.742667 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":108,"threads_started":1}
I20260927 16:41:21.744551 15053 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45119
I20260927 16:41:21.744771 15053 heartbeater.cc:464] Registering TS with master...
I20260927 16:41:21.745245 15053 heartbeater.cc:511] Master 127.0.98.62:45119 requested a full tablet report, sending...
I20260927 16:41:21.746214 14727 ts_manager.cc:194] Registered new tserver with Master: 48fe5a20c28142bca8ee7a378df77b86 (127.0.98.2:36877)
I20260927 16:41:21.746902 14727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36413
W20260927 16:41:21.839038 15056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:21.839322 15056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:21.839367 15056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:21.843173 15056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:21.843276 15056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:21.843353 15056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:41:21.848242 15056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39537
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42177
--tserver_master_addrs=127.0.98.62:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:21.849202 15056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:21.850112 15056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:41:21.859129 15064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:21.859125 15061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:21.859236 15056 server_base.cc:1034] running on GCE node
W20260927 16:41:21.859125 15062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:21.859727 15056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:21.860307 15056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:21.861464 15056 hybrid_clock.cc:648] HybridClock initialized: now 1790527281861452 us; error 22 us; skew 500 ppm
Sep 27 16:41:21 dist-test-slave-z40b krb5kdc[14146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:21.863955 15056 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:41:21.864763 15056 webserver.cc:460] Webserver started at http://127.0.98.3:42177/ using document root <none> and password file <none>
I20260927 16:41:21.865278 15056 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:21.865365 15056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:41:21.869174 15056 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 16:41:21.871083 15071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:21.872247 15056 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:41:21.872398 15056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8"
format_stamp: "Formatted at 2026-09-27 16:41:20 on dist-test-slave-z40b"
server_key: "620a1470d0fce7be2f68645d1eed3ab6"
server_key_iv: "fde9ec72ddbf9c6cad9fd2d7df8e3394"
server_key_version: "encryptionkey@0"
I20260927 16:41:21.872901 15056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:21.888113 15056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:21.890766 15056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:21.890945 15056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:21.891530 15056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:41:21.892273 15056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:41:21.892342 15056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:21.892410 15056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:41:21.892452 15056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:21.900944 15056 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39537
I20260927 16:41:21.900964 15184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39537 every 8 connection(s)
I20260927 16:41:21.901818 15056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:41:21.909548 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 15056
Sep 27 16:41:21 dist-test-slave-z40b krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:41:21.914827 14919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.903477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59295 (local address 127.0.98.62:45119)
0927 16:41:21.903598 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:21.903602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:21.904387 (+ 785us) server_negotiation.cc:407] Connection header received
0927 16:41:21.905599 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:21.905602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:21.905659 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:21.905735 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:21.906819 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.907430 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.908607 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.908851 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.908953 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.911283 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:21.911286 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:21.911288 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:21.911310 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:21.912850 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:21.913302 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:21.913307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:21.913308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:21.913386 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:21.913755 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:21.913760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:21.913765 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:21.913954 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:21.914022 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:21.914570 (+ 548us) server_negotiation.cc:299] Negotiation successful
0927 16:41:21.914686 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 16:41:21.915751 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.903774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:59295)
0927 16:41:21.904257 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:21.904294 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:21.905392 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:21.905912 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:21.905918 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:21.906262 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:41:21.906672 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.906683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.907547 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.907553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.908480 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.908492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.908985 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.908989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.909138 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.909675 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:41:21.909690 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:41:21.911089 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:41:21.912949 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:21.912954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:21.912966 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:21.913201 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:21.913508 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:21.913514 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:21.913516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:21.913668 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:21.914054 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:41:21.914061 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:41:21.914330 (+ 269us) client_negotiation.cc:769] Sending connection context
0927 16:41:21.914527 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 16:41:21.914881 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":114,"threads_started":1}
I20260927 16:41:21.916777 15185 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45119
I20260927 16:41:21.916993 15185 heartbeater.cc:464] Registering TS with master...
I20260927 16:41:21.917498 15185 heartbeater.cc:511] Master 127.0.98.62:45119 requested a full tablet report, sending...
I20260927 16:41:21.918215 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.910155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46188 (local address 127.0.98.62:45119)
0927 16:41:21.910693 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:21.910702 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:21.910760 (+ 58us) server_negotiation.cc:407] Connection header received
0927 16:41:21.910795 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:21.910797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:21.910846 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:21.910943 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:21.911788 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.912659 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.913468 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.913730 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.913820 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.915218 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:21.915224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:21.915228 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:21.915255 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:21.916597 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:21.917014 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:21.917017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:21.917018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:21.917113 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:21.917382 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:21.917383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:21.917385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:21.917520 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:21.917730 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:21.917906 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 16:41:21.918021 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"spinlock_wait_cycles":3840,"thread_start_us":83,"threads_started":1}
I20260927 16:41:21.918524 14727 ts_manager.cc:194] Registered new tserver with Master: 3b5a90dc9f9d41de9b4fabb7c5e21be8 (127.0.98.3:39537)
I20260927 16:41:21.919133 14727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59295
I20260927 16:41:21.919955 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:41:21.923751 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.920544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46196 (local address 127.0.98.62:45119)
0927 16:41:21.920641 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:21.920644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:21.920868 (+ 224us) server_negotiation.cc:407] Connection header received
0927 16:41:21.920989 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:21.920991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:21.921067 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:21.921140 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:41:21.921350 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.921797 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.922547 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.922745 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.922810 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.923021 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:41:21.923393 (+ 372us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:41:21.923491 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:21.923547 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 16:41:21.923587 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 16:41:21.924310 14726 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:46196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:41:21.927450 14726 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:41:21.934391 14849 tablet_service.cc:1467] Processing CreateTablet for tablet 3cb83a7c26aa49a6a7aea5fc0222871f (DEFAULT_TABLE table=test-table [id=cc348006eb2641a28868ae22b5d548ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:41:21.935011 14849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cb83a7c26aa49a6a7aea5fc0222871f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:41:21.937868 14807 tablet_bootstrap.cc:492] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105: Bootstrap starting.
I20260927 16:41:21.938488 14922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.933427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39537 (local address 127.0.0.1:57596)
0927 16:41:21.933537 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:21.933552 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:21.933639 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:21.934703 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:21.934706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:21.934720 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:41:21.934793 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.934797 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.936142 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.936144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.937323 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.937334 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.938134 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.938138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.938241 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.938284 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 16:41:21.938336 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 16:41:21.938403 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 16:41:21.938763 14807 tablet_bootstrap.cc:654] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105: Neither blocks nor log segments found. Creating new log.
I20260927 16:41:21.939178 15191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.933851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57596 (local address 127.0.98.3:39537)
0927 16:41:21.934189 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:21.934212 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:21.934237 (+ 25us) server_negotiation.cc:407] Connection header received
0927 16:41:21.934309 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:21.934314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:21.934521 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:21.934909 (+ 388us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:41:21.935013 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.936030 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.937443 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.937986 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.938752 (+ 766us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.938906 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:21.938954 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 16:41:21.939019 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":161,"threads_started":1}
I20260927 16:41:21.940367 14807 tablet_bootstrap.cc:492] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105: No bootstrap required, opened a new log
I20260927 16:41:21.940474 14807 ts_tablet_manager.cc:1397] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 16:41:21.940900 14807 raft_consensus.cc:348] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } }
I20260927 16:41:21.941021 14807 raft_consensus.cc:374] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:41:21.941057 14807 raft_consensus.cc:729] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbff7cfa13fc4c1ab07b892db9b7f105, State: Initialized, Role: FOLLOWER
I20260927 16:41:21.941208 14807 consensus_queue.cc:260] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } }
I20260927 16:41:21.941547 14807 ts_tablet_manager.cc:1428] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 16:41:21.942276 15119 tablet_service.cc:1467] Processing CreateTablet for tablet 3cb83a7c26aa49a6a7aea5fc0222871f (DEFAULT_TABLE table=test-table [id=cc348006eb2641a28868ae22b5d548ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:41:21.943135 15192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.934323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36877 (local address 127.0.0.1:49454)
0927 16:41:21.938056 (+ 3733us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:21.938070 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:21.938155 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:21.938862 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:21.938865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:21.938881 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:41:21.938970 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.938975 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.940799 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.940804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.941834 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.941844 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.942673 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.942677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.942777 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.942799 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 16:41:21.942927 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 16:41:21.943004 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3659,"thread_start_us":58,"threads_started":1}
I20260927 16:41:21.943451 15119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cb83a7c26aa49a6a7aea5fc0222871f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:41:21.943730 15193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.934509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49454 (local address 127.0.98.2:36877)
0927 16:41:21.934889 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:21.934899 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:21.938261 (+ 3362us) server_negotiation.cc:407] Connection header received
0927 16:41:21.938340 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:21.938345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:21.938531 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:21.938677 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:41:21.939099 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.940661 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.941997 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.942551 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.942674 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.942807 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:21.943458 (+ 651us) server_negotiation.cc:299] Negotiation successful
0927 16:41:21.943567 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":85,"threads_started":1}
I20260927 16:41:21.945752 14987 tablet_service.cc:1467] Processing CreateTablet for tablet 3cb83a7c26aa49a6a7aea5fc0222871f (DEFAULT_TABLE table=test-table [id=cc348006eb2641a28868ae22b5d548ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:41:21.946533 14987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cb83a7c26aa49a6a7aea5fc0222871f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:41:21.949846 15195 tablet_bootstrap.cc:492] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8: Bootstrap starting.
I20260927 16:41:21.951874 15195 tablet_bootstrap.cc:654] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8: Neither blocks nor log segments found. Creating new log.
I20260927 16:41:21.952530 15196 tablet_bootstrap.cc:492] T 3cb83a7c26aa49a6a7aea5fc0222871f P 48fe5a20c28142bca8ee7a378df77b86: Bootstrap starting.
I20260927 16:41:21.952962 15195 log.cc:824] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8: Log is configured to *not* fsync() on all Append() calls
I20260927 16:41:21.954420 15196 tablet_bootstrap.cc:654] T 3cb83a7c26aa49a6a7aea5fc0222871f P 48fe5a20c28142bca8ee7a378df77b86: Neither blocks nor log segments found. Creating new log.
I20260927 16:41:21.954594 15195 tablet_bootstrap.cc:492] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8: No bootstrap required, opened a new log
I20260927 16:41:21.954782 15195 ts_tablet_manager.cc:1397] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:41:21.955121 15196 log.cc:824] T 3cb83a7c26aa49a6a7aea5fc0222871f P 48fe5a20c28142bca8ee7a378df77b86: Log is configured to *not* fsync() on all Append() calls
I20260927 16:41:21.956799 15196 tablet_bootstrap.cc:492] T 3cb83a7c26aa49a6a7aea5fc0222871f P 48fe5a20c28142bca8ee7a378df77b86: No bootstrap required, opened a new log
I20260927 16:41:21.956971 15196 ts_tablet_manager.cc:1397] T 3cb83a7c26aa49a6a7aea5fc0222871f P 48fe5a20c28142bca8ee7a378df77b86: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 16:41:21.957858 15195 raft_consensus.cc:348] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } }
I20260927 16:41:21.958079 15195 raft_consensus.cc:374] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:41:21.958135 15195 raft_consensus.cc:729] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5a90dc9f9d41de9b4fabb7c5e21be8, State: Initialized, Role: FOLLOWER
I20260927 16:41:21.958526 15195 consensus_queue.cc:260] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } }
I20260927 16:41:21.959087 15185 heartbeater.cc:503] Master 127.0.98.62:45119 was elected leader, sending a full tablet report...
I20260927 16:41:21.959420 15195 ts_tablet_manager.cc:1428] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 16:41:21.959722 15196 raft_consensus.cc:348] T 3cb83a7c26aa49a6a7aea5fc0222871f P 48fe5a20c28142bca8ee7a378df77b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } }
I20260927 16:41:21.959955 15196 raft_consensus.cc:374] T 3cb83a7c26aa49a6a7aea5fc0222871f P 48fe5a20c28142bca8ee7a378df77b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:41:21.960007 15196 raft_consensus.cc:729] T 3cb83a7c26aa49a6a7aea5fc0222871f P 48fe5a20c28142bca8ee7a378df77b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48fe5a20c28142bca8ee7a378df77b86, State: Initialized, Role: FOLLOWER
I20260927 16:41:21.960397 15196 consensus_queue.cc:260] T 3cb83a7c26aa49a6a7aea5fc0222871f P 48fe5a20c28142bca8ee7a378df77b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } }
I20260927 16:41:21.961011 15053 heartbeater.cc:503] Master 127.0.98.62:45119 was elected leader, sending a full tablet report...
I20260927 16:41:21.961349 15196 ts_tablet_manager.cc:1428] T 3cb83a7c26aa49a6a7aea5fc0222871f P 48fe5a20c28142bca8ee7a378df77b86: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 16:41:21.978966 15199 raft_consensus.cc:482] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:41:21.979123 15199 raft_consensus.cc:504] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } }
I20260927 16:41:21.980039 15199 leader_election.cc:288] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48fe5a20c28142bca8ee7a378df77b86 (127.0.98.2:36877), bbff7cfa13fc4c1ab07b892db9b7f105 (127.0.98.1:33767)
W20260927 16:41:21.981204 15054 tablet.cc:2369] T 3cb83a7c26aa49a6a7aea5fc0222871f P 48fe5a20c28142bca8ee7a378df77b86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:41:21.984014 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.980234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36877 (local address 127.0.98.3:50203)
0927 16:41:21.980361 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:21.980378 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:21.980461 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:21.980699 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:21.980702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:21.980720 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:41:21.980857 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.980867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.981835 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.981837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.982941 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.982950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.983701 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.983704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.983776 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.983793 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 16:41:21.983849 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 16:41:21.983898 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 16:41:21.984068 15193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.980336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50203 (local address 127.0.98.2:36877)
0927 16:41:21.980459 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:21.980463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:21.980479 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:41:21.980526 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:21.980530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:21.980578 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:21.980664 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:41:21.980969 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.981713 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.983058 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.983591 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.983693 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.983756 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:21.983886 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 16:41:21.983959 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:41:21.984468 14923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.980342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41903 (local address 127.0.98.1:33767)
0927 16:41:21.980479 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:21.980483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:21.980934 (+ 451us) server_negotiation.cc:407] Connection header received
0927 16:41:21.980995 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:21.980998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:21.981056 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:21.981147 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:41:21.981418 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.982068 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.983281 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.983907 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.984002 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.984132 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:21.984283 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 16:41:21.984355 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 16:41:21.984524 15007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cb83a7c26aa49a6a7aea5fc0222871f" candidate_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48fe5a20c28142bca8ee7a378df77b86" is_pre_election: true
I20260927 16:41:21.984812 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:21.980242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33767 (local address 127.0.98.3:41903)
0927 16:41:21.980821 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:21.980834 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:21.980927 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:21.981181 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:21.981186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:21.981206 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:41:21.981318 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.981325 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.982174 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.982178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.983183 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:21.983189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:21.984040 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:21.984045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:21.984139 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:21.984164 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 16:41:21.984618 (+ 454us) client_negotiation.cc:240] Negotiation successful
0927 16:41:21.984702 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":252,"thread_start_us":112,"threads_started":1}
I20260927 16:41:21.984802 15007 raft_consensus.cc:2454] T 3cb83a7c26aa49a6a7aea5fc0222871f P 48fe5a20c28142bca8ee7a378df77b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5a90dc9f9d41de9b4fabb7c5e21be8 in term 0.
I20260927 16:41:21.985280 15073 leader_election.cc:302] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5a90dc9f9d41de9b4fabb7c5e21be8, 48fe5a20c28142bca8ee7a378df77b86; no voters:
I20260927 16:41:21.985352 14869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cb83a7c26aa49a6a7aea5fc0222871f" candidate_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" is_pre_election: true
I20260927 16:41:21.985523 15199 raft_consensus.cc:2781] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:41:21.985603 15199 raft_consensus.cc:482] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:41:21.985618 14869 raft_consensus.cc:2454] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b5a90dc9f9d41de9b4fabb7c5e21be8 in term 0.
I20260927 16:41:21.985657 15199 raft_consensus.cc:3037] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:41:21.986572 15199 raft_consensus.cc:504] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } }
I20260927 16:41:21.986938 15199 leader_election.cc:288] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [CANDIDATE]: Term 1 election: Requested vote from peers 48fe5a20c28142bca8ee7a378df77b86 (127.0.98.2:36877), bbff7cfa13fc4c1ab07b892db9b7f105 (127.0.98.1:33767)
I20260927 16:41:21.987308 15007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cb83a7c26aa49a6a7aea5fc0222871f" candidate_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48fe5a20c28142bca8ee7a378df77b86"
I20260927 16:41:21.987306 14869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cb83a7c26aa49a6a7aea5fc0222871f" candidate_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105"
I20260927 16:41:21.987421 14869 raft_consensus.cc:3037] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:41:21.987421 15007 raft_consensus.cc:3037] T 3cb83a7c26aa49a6a7aea5fc0222871f P 48fe5a20c28142bca8ee7a378df77b86 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:41:21.988612 15007 raft_consensus.cc:2454] T 3cb83a7c26aa49a6a7aea5fc0222871f P 48fe5a20c28142bca8ee7a378df77b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5a90dc9f9d41de9b4fabb7c5e21be8 in term 1.
I20260927 16:41:21.988612 14869 raft_consensus.cc:2454] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b5a90dc9f9d41de9b4fabb7c5e21be8 in term 1.
I20260927 16:41:21.988868 15073 leader_election.cc:302] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5a90dc9f9d41de9b4fabb7c5e21be8, 48fe5a20c28142bca8ee7a378df77b86; no voters:
I20260927 16:41:21.989028 15199 raft_consensus.cc:2781] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:41:21.989233 15199 raft_consensus.cc:686] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [term 1 LEADER]: Becoming Leader. State: Replica: 3b5a90dc9f9d41de9b4fabb7c5e21be8, State: Running, Role: LEADER
I20260927 16:41:21.989512 15199 consensus_queue.cc:237] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } }
I20260927 16:41:21.992316 14725 catalog_manager.cc:5495] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b5a90dc9f9d41de9b4fabb7c5e21be8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } health_report { overall_health: UNKNOWN } } }
I20260927 16:41:22.006826 15191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:22.003982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57608 (local address 127.0.98.3:39537)
0927 16:41:22.004102 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:22.004105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:22.004120 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:41:22.004248 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:22.004250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:22.004305 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:22.004383 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:41:22.004545 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:22.004980 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:22.005690 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:22.005864 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:22.005928 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:22.006164 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:41:22.006507 (+ 343us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:41:22.006588 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:22.006640 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 16:41:22.006681 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 16:41:22.011945 15007 raft_consensus.cc:1261] T 3cb83a7c26aa49a6a7aea5fc0222871f P 48fe5a20c28142bca8ee7a378df77b86 [term 1 FOLLOWER]: Refusing update from remote peer 3b5a90dc9f9d41de9b4fabb7c5e21be8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:41:22.011967 14869 raft_consensus.cc:1261] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [term 1 FOLLOWER]: Refusing update from remote peer 3b5a90dc9f9d41de9b4fabb7c5e21be8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:41:22.012619 15199 consensus_queue.cc:1035] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:41:22.012835 15202 consensus_queue.cc:1035] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:41:22.019336 15207 mvcc.cc:205] Tried to move back new op lower bound from 7333999747114889216 to 7333999747031351296. Current Snapshot: MvccSnapshot[applied={T|T < 7333999747114889216}]
I20260927 16:41:22.025736 14726 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:46196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:41:22.025879 14726 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:46196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:41:22.028034 14726 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58: Sending DeleteTablet for 3 replicas of tablet 3cb83a7c26aa49a6a7aea5fc0222871f
I20260927 16:41:22.028631 15119 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cb83a7c26aa49a6a7aea5fc0222871f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:41:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57596
I20260927 16:41:22.028689 14849 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cb83a7c26aa49a6a7aea5fc0222871f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:41:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45180
I20260927 16:41:22.028859 14924 tablet_replica.cc:330] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105: stopping tablet replica
I20260927 16:41:22.028823 14987 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cb83a7c26aa49a6a7aea5fc0222871f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:41:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49454
I20260927 16:41:22.028955 14924 raft_consensus.cc:2229] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:41:22.029070 14924 raft_consensus.cc:2258] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:41:22.029233 15216 tablet_replica.cc:330] T 3cb83a7c26aa49a6a7aea5fc0222871f P 48fe5a20c28142bca8ee7a378df77b86: stopping tablet replica
I20260927 16:41:22.029453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 14765
I20260927 16:41:22.029521 15216 raft_consensus.cc:2229] T 3cb83a7c26aa49a6a7aea5fc0222871f P 48fe5a20c28142bca8ee7a378df77b86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:41:22.030176 15215 tablet_replica.cc:330] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8: stopping tablet replica
I20260927 16:41:22.030251 15216 raft_consensus.cc:2258] T 3cb83a7c26aa49a6a7aea5fc0222871f P 48fe5a20c28142bca8ee7a378df77b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:41:22.030464 15215 raft_consensus.cc:2229] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:41:22.030745 15215 raft_consensus.cc:2258] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:41:22.031643 15216 ts_tablet_manager.cc:1905] T 3cb83a7c26aa49a6a7aea5fc0222871f P 48fe5a20c28142bca8ee7a378df77b86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:41:22.031793 15215 ts_tablet_manager.cc:1905] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:41:22.033785 15215 ts_tablet_manager.cc:1918] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:41:22.033891 15215 log.cc:1196] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/wal/wals/3cb83a7c26aa49a6a7aea5fc0222871f
I20260927 16:41:22.034188 15215 ts_tablet_manager.cc:1939] T 3cb83a7c26aa49a6a7aea5fc0222871f P 3b5a90dc9f9d41de9b4fabb7c5e21be8: Deleting consensus metadata
I20260927 16:41:22.034904 15216 ts_tablet_manager.cc:1918] T 3cb83a7c26aa49a6a7aea5fc0222871f P 48fe5a20c28142bca8ee7a378df77b86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:41:22.035006 15216 log.cc:1196] T 3cb83a7c26aa49a6a7aea5fc0222871f P 48fe5a20c28142bca8ee7a378df77b86: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/wal/wals/3cb83a7c26aa49a6a7aea5fc0222871f
I20260927 16:41:22.035074 14714 catalog_manager.cc:4841] TS 3b5a90dc9f9d41de9b4fabb7c5e21be8 (127.0.98.3:39537): tablet 3cb83a7c26aa49a6a7aea5fc0222871f (table test-table [id=cc348006eb2641a28868ae22b5d548ca]) successfully deleted
I20260927 16:41:22.035347 15216 ts_tablet_manager.cc:1939] T 3cb83a7c26aa49a6a7aea5fc0222871f P 48fe5a20c28142bca8ee7a378df77b86: Deleting consensus metadata
I20260927 16:41:22.036404 14712 catalog_manager.cc:4841] TS 48fe5a20c28142bca8ee7a378df77b86 (127.0.98.2:36877): tablet 3cb83a7c26aa49a6a7aea5fc0222871f (table test-table [id=cc348006eb2641a28868ae22b5d548ca]) successfully deleted
W20260927 16:41:22.037732 14714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33767 (error 108)
I20260927 16:41:22.038201 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 14918
I20260927 16:41:22.038981 15192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:22.038350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33767 (local address 127.0.0.1:45196)
0927 16:41:22.038842 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:22.038905 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424}
W20260927 16:41:22.040023 14714 catalog_manager.cc:4584] TS bbff7cfa13fc4c1ab07b892db9b7f105 (127.0.98.1:33767): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3cb83a7c26aa49a6a7aea5fc0222871f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33767: connect: Connection refused (error 111)
I20260927 16:41:22.045027 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 15056
I20260927 16:41:22.049700 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 14694
I20260927 16:41:22.055604 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45119
--webserver_interface=127.0.98.62
--webserver_port=39371
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45119
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:41:22.156853 15218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:22.157083 15218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:22.157124 15218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:22.160982 15218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:41:22.161026 15218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:22.161046 15218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:22.161062 15218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:41:22.161121 15218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:41:22.165735 15218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:22.166604 15218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:22.167557 15218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:41:22.175045 15226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:22.175069 15224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:22.175136 15223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:22.175444 15218 server_base.cc:1034] running on GCE node
I20260927 16:41:22.175863 15218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:22.176750 15218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:22.177898 15218 hybrid_clock.cc:648] HybridClock initialized: now 1790527282177883 us; error 26 us; skew 500 ppm
Sep 27 16:41:22 dist-test-slave-z40b krb5kdc[14146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:22.180169 15218 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:41:22.180874 15218 webserver.cc:460] Webserver started at http://127.0.98.62:39371/ using document root <none> and password file <none>
I20260927 16:41:22.181326 15218 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:22.181381 15218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:41:22.184674 15218 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260927 16:41:22.186439 15233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:22.187597 15218 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:41:22.187726 15218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "ed559add211b4fe190913711a83ace58"
format_stamp: "Formatted at 2026-09-27 16:41:20 on dist-test-slave-z40b"
server_key: "7dcc356e390aaab0e210d29dd3a38588"
server_key_iv: "543efba9c1852c62580d11adf0a57e14"
server_key_version: "encryptionkey@0"
I20260927 16:41:22.188191 15218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:22.221575 15218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:22.224651 15218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:22.224828 15218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:22.230978 15218 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45119
I20260927 16:41:22.231314 15285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45119 every 8 connection(s)
I20260927 16:41:22.232044 15218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:41:22.235873 15286 sys_catalog.cc:263] Verifying existing consensus state
I20260927 16:41:22.237323 15286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58: Bootstrap starting.
I20260927 16:41:22.241552 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 15218
I20260927 16:41:22.241948 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33767
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45119
--webserver_port=33135
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:41:22.254009 15286 log.cc:824] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58: Log is configured to *not* fsync() on all Append() calls
I20260927 16:41:22.264639 15286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:41:22.265008 15286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58: Bootstrap complete.
I20260927 16:41:22.268860 15286 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed559add211b4fe190913711a83ace58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } }
I20260927 16:41:22.269198 15286 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed559add211b4fe190913711a83ace58, State: Initialized, Role: FOLLOWER
I20260927 16:41:22.269692 15286 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed559add211b4fe190913711a83ace58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } }
I20260927 16:41:22.269809 15286 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:41:22.269892 15286 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:41:22.270012 15286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [term 2 FOLLOWER]: Advancing to term 3
I20260927 16:41:22.271859 15286 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed559add211b4fe190913711a83ace58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } }
I20260927 16:41:22.272176 15286 leader_election.cc:302] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed559add211b4fe190913711a83ace58; no voters:
I20260927 16:41:22.272483 15286 leader_election.cc:288] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 16:41:22.272557 15290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 16:41:22.272734 15290 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [term 3 LEADER]: Becoming Leader. State: Replica: ed559add211b4fe190913711a83ace58, State: Running, Role: LEADER
I20260927 16:41:22.273000 15290 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed559add211b4fe190913711a83ace58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } }
I20260927 16:41:22.273586 15286 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:41:22.273828 15290 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed559add211b4fe190913711a83ace58. Latest consensus state: current_term: 3 leader_uuid: "ed559add211b4fe190913711a83ace58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed559add211b4fe190913711a83ace58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } } }
I20260927 16:41:22.274024 15290 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [sys.catalog]: This master's current role is: LEADER
I20260927 16:41:22.273814 15291 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ed559add211b4fe190913711a83ace58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed559add211b4fe190913711a83ace58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45119 } } }
I20260927 16:41:22.274387 15291 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed559add211b4fe190913711a83ace58 [sys.catalog]: This master's current role is: LEADER
I20260927 16:41:22.275151 15294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:41:22.279867 15294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:41:22.280484 15294 catalog_manager.cc:1259] Loaded cluster ID: 06701aaea5784b5385b3207bbd6a9839
I20260927 16:41:22.280541 15294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 16:41:22.281525 15294 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbe734ac9ad google::LogMessage::Fail() at ??:0
@ 0x7fbe734af88a google::LogMessage::SendToLog() at ??:0
@ 0x7fbe734ac4de google::LogMessage::Flush() at ??:0
@ 0x7fbe734ada19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbe7ddebfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fbe7ddecd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbe7dde9579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fbe7de2d640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fbe756fe3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbe756fec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fbe757003f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fbe756ef512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbe76d336db start_thread at ??:0
@ 0x7fbe71faa61f clone at ??:0
W20260927 16:41:22.374670 15288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:22.374959 15288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:22.375010 15288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:22.380430 15288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:22.380499 15288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:22.380586 15288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:41:22.387918 15288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33767
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33135
--tserver_master_addrs=127.0.98.62:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:22.389133 15288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:22.390386 15288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:41:22.400987 15328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:22.401011 15330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:22.400981 15327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:22.401921 15288 server_base.cc:1034] running on GCE node
I20260927 16:41:22.402292 15288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:22.402869 15288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:22.404026 15288 hybrid_clock.cc:648] HybridClock initialized: now 1790527282403999 us; error 40 us; skew 500 ppm
Sep 27 16:41:22 dist-test-slave-z40b krb5kdc[14146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:22.406617 15288 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:41:22.407558 15288 webserver.cc:460] Webserver started at http://127.0.98.1:33135/ using document root <none> and password file <none>
I20260927 16:41:22.408077 15288 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:22.408135 15288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:41:22.411758 15288 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:41:22.413841 15337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:22.414978 15288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:41:22.415078 15288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "bbff7cfa13fc4c1ab07b892db9b7f105"
format_stamp: "Formatted at 2026-09-27 16:41:20 on dist-test-slave-z40b"
server_key: "e017c70c57309ce7d4c9835237eee7e6"
server_key_iv: "abe5007fb5b487bc4768842fe1a91804"
server_key_version: "encryptionkey@0"
I20260927 16:41:22.415657 15288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:22.424885 15288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:22.428043 15288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:22.428220 15288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:22.428714 15288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:41:22.429651 15344 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 16:41:22.434686 15288 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 16:41:22.434769 15288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 16:41:22.434823 15288 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 16:41:22.436620 15288 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 16:41:22.436684 15288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260927 16:41:22.436939 15344 tablet_bootstrap.cc:492] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105: Bootstrap starting.
I20260927 16:41:22.446762 15288 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33767
I20260927 16:41:22.446815 15451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33767 every 8 connection(s)
I20260927 16:41:22.447731 15288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:41:22.448256 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 15288
I20260927 16:41:22.448728 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36877
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45119
--webserver_port=39323
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:41:22.457366 15344 log.cc:824] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105: Log is configured to *not* fsync() on all Append() calls
I20260927 16:41:22.465390 15344 tablet_bootstrap.cc:492] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:41:22.465747 15344 tablet_bootstrap.cc:492] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105: Bootstrap complete.
I20260927 16:41:22.466825 15344 ts_tablet_manager.cc:1397] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105: Time spent bootstrapping tablet: real 0.030s user 0.026s sys 0.000s
I20260927 16:41:22.469837 15344 raft_consensus.cc:348] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } }
I20260927 16:41:22.470337 15344 raft_consensus.cc:729] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbff7cfa13fc4c1ab07b892db9b7f105, State: Initialized, Role: FOLLOWER
I20260927 16:41:22.470824 15344 consensus_queue.cc:260] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } }
I20260927 16:41:22.471817 15344 ts_tablet_manager.cc:1428] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260927 16:41:22.515084 15455 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:41:22.450595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:35319)
0927 16:41:22.451789 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:22.451829 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:22.452503 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:22.514310 (+ 61807us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1018,"thread_start_us":185,"threads_started":1}
W20260927 16:41:22.515647 15452 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45119 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45119: Client connection negotiation failed: client connection to 127.0.98.62:45119: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 16:41:22.516353 15455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:22.516098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:54757)
0927 16:41:22.516222 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:22.516303 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 16:41:22.517031 15455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:22.516849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:36783)
0927 16:41:22.516945 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:22.516985 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 16:41:22.517273 15452 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 16:41:22.554579 15454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:22.554771 15454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:22.554807 15454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:22.558284 15454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:22.558324 15454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:22.558374 15454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:41:22.563055 15454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39323
--tserver_master_addrs=127.0.98.62:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:22.563942 15454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:22.564864 15454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:41:22.573431 15465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:22.573483 15463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:22.573639 15454 server_base.cc:1034] running on GCE node
W20260927 16:41:22.573690 15462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:22.574030 15454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:22.574597 15454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:22.575743 15454 hybrid_clock.cc:648] HybridClock initialized: now 1790527282575733 us; error 23 us; skew 500 ppm
Sep 27 16:41:22 dist-test-slave-z40b krb5kdc[14146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:22.578291 15454 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:41:22.579011 15454 webserver.cc:460] Webserver started at http://127.0.98.2:39323/ using document root <none> and password file <none>
I20260927 16:41:22.579545 15454 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:22.579612 15454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:41:22.582846 15454 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 16:41:22.584632 15472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:22.585748 15454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:41:22.585871 15454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "48fe5a20c28142bca8ee7a378df77b86"
format_stamp: "Formatted at 2026-09-27 16:41:20 on dist-test-slave-z40b"
server_key: "d61f39e79df0a36ac517dc486f7baa7d"
server_key_iv: "aaef5b44e3fa8535c4f6bf7030388f7d"
server_key_version: "encryptionkey@0"
I20260927 16:41:22.586349 15454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:22.597651 15454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:22.600112 15454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:22.600282 15454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:22.600788 15454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:41:22.601524 15454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:41:22.601593 15454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:22.601656 15454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:41:22.601712 15454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:22.610232 15454 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36877
I20260927 16:41:22.610245 15585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36877 every 8 connection(s)
I20260927 16:41:22.611106 15454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:41:22.614454 15588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:22.612940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:37489)
0927 16:41:22.613333 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:22.613425 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":112,"threads_started":1}
W20260927 16:41:22.614866 15586 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45119 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45119: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
I20260927 16:41:22.615396 15588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:22.615162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:59589)
0927 16:41:22.615306 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:22.615347 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 16:41:22.615949 15588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:22.615774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:59441)
0927 16:41:22.615870 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:22.615907 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 16:41:22.616130 15586 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 16:41:22.616258 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 15454
I20260927 16:41:22.616618 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39537
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45119
--webserver_port=42177
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:41:22.717717 15589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:22.717938 15589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:22.717978 15589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:22.721946 15589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:22.721992 15589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:22.722047 15589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:41:22.726818 15589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39537
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42177
--tserver_master_addrs=127.0.98.62:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:22.727666 15589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:22.728579 15589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:41:22.737118 15597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:22.737136 15594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:22.737165 15589 server_base.cc:1034] running on GCE node
W20260927 16:41:22.737121 15595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:22.737615 15589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:22.738193 15589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:22.739382 15589 hybrid_clock.cc:648] HybridClock initialized: now 1790527282739350 us; error 42 us; skew 500 ppm
Sep 27 16:41:22 dist-test-slave-z40b krb5kdc[14146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:22.741942 15589 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:41:22.742697 15589 webserver.cc:460] Webserver started at http://127.0.98.3:42177/ using document root <none> and password file <none>
I20260927 16:41:22.743197 15589 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:22.743341 15589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:41:22.746764 15589 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:41:22.748749 15604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:22.749807 15589 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:41:22.749948 15589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8"
format_stamp: "Formatted at 2026-09-27 16:41:20 on dist-test-slave-z40b"
server_key: "620a1470d0fce7be2f68645d1eed3ab6"
server_key_iv: "fde9ec72ddbf9c6cad9fd2d7df8e3394"
server_key_version: "encryptionkey@0"
I20260927 16:41:22.750427 15589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:22.765540 15589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:22.768198 15589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:22.768388 15589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:22.768937 15589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:41:22.769654 15589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:41:22.769721 15589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:22.769785 15589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:41:22.769833 15589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:22.777846 15589 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39537
I20260927 16:41:22.777908 15717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39537 every 8 connection(s)
I20260927 16:41:22.778765 15589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:41:22.782019 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 15589
I20260927 16:41:22.782317 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:22.780670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:59587)
0927 16:41:22.781087 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:22.781183 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":127,"threads_started":1}
W20260927 16:41:22.782796 15718 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45119 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45119: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
I20260927 16:41:22.783370 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:22.783141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:37457)
0927 16:41:22.783275 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:22.783317 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 16:41:22.783936 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:22.783779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:47877)
0927 16:41:22.783860 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:22.783896 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 16:41:22.784125 15718 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 16:41:23.518260 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:23.517858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:38843)
0927 16:41:23.518116 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:23.518163 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":66,"threads_started":1}
I20260927 16:41:23.617053 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:23.616585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:37159)
0927 16:41:23.616901 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:23.616967 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":72,"threads_started":1}
I20260927 16:41:23.676946 15724 raft_consensus.cc:482] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:41:23.677117 15724 raft_consensus.cc:504] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } }
I20260927 16:41:23.678092 15724 leader_election.cc:288] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3b5a90dc9f9d41de9b4fabb7c5e21be8 (127.0.98.3:39537), 48fe5a20c28142bca8ee7a378df77b86 (127.0.98.2:36877)
Sep 27 16:41:23 dist-test-slave-z40b krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 27 16:41:23 dist-test-slave-z40b krb5kdc[14146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 16:41:23.694278 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:23.678279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36877 (local address 127.0.98.1:38033)
0927 16:41:23.678544 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:23.678559 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:23.678661 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:23.680652 (+ 1991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:23.680659 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:23.681318 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:41:23.682227 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:23.682246 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:23.684107 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:23.684110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:23.685798 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:23.685810 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:23.686970 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:23.686973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:23.687329 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:23.688027 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:41:23.688053 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:41:23.689969 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:41:23.692055 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:23.692061 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:23.692073 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:23.692326 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:23.692670 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:23.692672 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:23.692674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:23.692812 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:23.693365 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:41:23.693370 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:41:23.693600 (+ 230us) client_negotiation.cc:769] Sending connection context
0927 16:41:23.693781 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 16:41:23.694097 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":429,"spinlock_wait_cycles":23936}
I20260927 16:41:23.694329 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:23.678467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38033 (local address 127.0.98.2:36877)
0927 16:41:23.678833 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:23.678852 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:23.678892 (+ 40us) server_negotiation.cc:407] Connection header received
0927 16:41:23.679748 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:23.679754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:23.680189 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:23.680538 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:23.682476 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:23.683967 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:23.685939 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:23.686635 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:23.686796 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:23.690150 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:23.690155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:23.690165 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:23.690189 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:23.691922 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:23.692438 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:23.692461 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:23.692465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:23.692565 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:23.692900 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:23.692904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:23.692906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:23.693250 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:23.693375 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:23.693900 (+ 525us) server_negotiation.cc:299] Negotiation successful
0927 16:41:23.694153 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":77,"threads_started":1}
I20260927 16:41:23.694361 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:23.678455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40493 (local address 127.0.98.3:39537)
0927 16:41:23.678833 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:23.678849 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:23.678892 (+ 43us) server_negotiation.cc:407] Connection header received
0927 16:41:23.679739 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:23.679748 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:23.680189 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:23.680538 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:23.682476 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:23.683967 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:23.685939 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:23.686635 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:23.686796 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:23.690420 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:23.690425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:23.690435 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:23.690458 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:23.692283 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:23.692697 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:23.692702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:23.692713 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:23.692816 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:23.693082 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:23.693086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:23.693087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:23.693425 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:23.693548 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:23.693894 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 16:41:23.694177 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":65,"threads_started":1}
I20260927 16:41:23.694391 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:23.678279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39537 (local address 127.0.98.1:40493)
0927 16:41:23.678544 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:23.678558 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:23.678688 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:23.680652 (+ 1964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:23.680659 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:23.681318 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:41:23.682227 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:23.682246 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:23.684107 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:23.684110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:23.685798 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:23.685810 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:23.686970 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:23.686973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:23.687331 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:23.688033 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:41:23.688051 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:41:23.690235 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:41:23.692397 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:23.692401 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:23.692404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:23.692596 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:23.692909 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:23.692911 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:23.692913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:23.692991 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:23.693535 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:41:23.693538 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:41:23.693638 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 16:41:23.693775 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 16:41:23.694183 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":597,"spinlock_wait_cycles":17920,"thread_start_us":90,"threads_started":1}
W20260927 16:41:23.697280 15341 leader_election.cc:341] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3b5a90dc9f9d41de9b4fabb7c5e21be8 (127.0.98.3:39537): Not found: Tablet not found: 3cb83a7c26aa49a6a7aea5fc0222871f
W20260927 16:41:23.697443 15339 leader_election.cc:341] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48fe5a20c28142bca8ee7a378df77b86 (127.0.98.2:36877): Not found: Tablet not found: 3cb83a7c26aa49a6a7aea5fc0222871f
I20260927 16:41:23.697499 15341 leader_election.cc:302] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbff7cfa13fc4c1ab07b892db9b7f105; no voters: 3b5a90dc9f9d41de9b4fabb7c5e21be8, 48fe5a20c28142bca8ee7a378df77b86
I20260927 16:41:23.697762 15724 raft_consensus.cc:2726] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:41:23.784888 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:23.784551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:34389)
0927 16:41:23.784768 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:23.784812 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":75,"threads_started":1}
I20260927 16:41:24.519438 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:24.518999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:43437)
0927 16:41:24.519276 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:24.519336 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":69,"threads_started":1}
I20260927 16:41:24.617956 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:24.617640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:36423)
0927 16:41:24.617840 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:24.617885 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":64,"threads_started":1}
I20260927 16:41:24.785817 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:24.785468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:41519)
0927 16:41:24.785678 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:24.785734 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":67,"threads_started":1}
I20260927 16:41:25.520514 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:25.520146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:40675)
0927 16:41:25.520372 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:25.520423 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":78,"threads_started":1}
I20260927 16:41:25.619007 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:25.618651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:54595)
0927 16:41:25.618884 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:25.618927 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":103,"threads_started":1}
I20260927 16:41:25.786862 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:25.786477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:55097)
0927 16:41:25.786726 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:25.786775 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":71,"threads_started":1}
I20260927 16:41:25.883921 15735 raft_consensus.cc:482] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:41:25.884058 15735 raft_consensus.cc:504] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } }
I20260927 16:41:25.884387 15735 leader_election.cc:288] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3b5a90dc9f9d41de9b4fabb7c5e21be8 (127.0.98.3:39537), 48fe5a20c28142bca8ee7a378df77b86 (127.0.98.2:36877)
W20260927 16:41:25.885226 15339 leader_election.cc:341] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48fe5a20c28142bca8ee7a378df77b86 (127.0.98.2:36877): Not found: Tablet not found: 3cb83a7c26aa49a6a7aea5fc0222871f
W20260927 16:41:25.885314 15341 leader_election.cc:341] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3b5a90dc9f9d41de9b4fabb7c5e21be8 (127.0.98.3:39537): Not found: Tablet not found: 3cb83a7c26aa49a6a7aea5fc0222871f
I20260927 16:41:25.885358 15339 leader_election.cc:302] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbff7cfa13fc4c1ab07b892db9b7f105; no voters: 3b5a90dc9f9d41de9b4fabb7c5e21be8, 48fe5a20c28142bca8ee7a378df77b86
I20260927 16:41:25.885529 15735 raft_consensus.cc:2726] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:41:26.521575 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:26.521200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:54863)
0927 16:41:26.521432 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:26.521483 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":72,"threads_started":1}
I20260927 16:41:26.620178 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:26.619694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:33421)
0927 16:41:26.619997 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:26.620067 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":92,"threads_started":1}
I20260927 16:41:26.788051 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:26.787648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:50791)
0927 16:41:26.787916 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:26.787976 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":125,"threads_started":1}
I20260927 16:41:27.523006 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:27.522548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:59293)
0927 16:41:27.522851 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:27.522914 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":90,"threads_started":1}
I20260927 16:41:27.621398 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:27.620982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:40461)
0927 16:41:27.621251 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:27.621312 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":87,"threads_started":1}
I20260927 16:41:27.789178 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:27.788811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:60927)
0927 16:41:27.789042 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:27.789096 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":67,"threads_started":1}
I20260927 16:41:28.524037 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:28.523686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:46947)
0927 16:41:28.523902 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:28.523949 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":74,"threads_started":1}
I20260927 16:41:28.622449 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:28.622055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:40335)
0927 16:41:28.622302 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:28.622360 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":65,"threads_started":1}
I20260927 16:41:28.790341 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:28.789922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:52681)
0927 16:41:28.790213 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:28.790260 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":123,"threads_started":1}
I20260927 16:41:28.838477 15745 raft_consensus.cc:482] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:41:28.838603 15745 raft_consensus.cc:504] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } }
I20260927 16:41:28.839092 15745 leader_election.cc:288] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3b5a90dc9f9d41de9b4fabb7c5e21be8 (127.0.98.3:39537), 48fe5a20c28142bca8ee7a378df77b86 (127.0.98.2:36877)
W20260927 16:41:28.839910 15341 leader_election.cc:341] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3b5a90dc9f9d41de9b4fabb7c5e21be8 (127.0.98.3:39537): Not found: Tablet not found: 3cb83a7c26aa49a6a7aea5fc0222871f
W20260927 16:41:28.839996 15339 leader_election.cc:341] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48fe5a20c28142bca8ee7a378df77b86 (127.0.98.2:36877): Not found: Tablet not found: 3cb83a7c26aa49a6a7aea5fc0222871f
I20260927 16:41:28.840085 15339 leader_election.cc:302] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbff7cfa13fc4c1ab07b892db9b7f105; no voters: 3b5a90dc9f9d41de9b4fabb7c5e21be8, 48fe5a20c28142bca8ee7a378df77b86
I20260927 16:41:28.840233 15745 raft_consensus.cc:2726] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:41:29.525110 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:29.524733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:38067)
0927 16:41:29.524967 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:29.525016 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
I20260927 16:41:29.623492 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:29.623096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:39903)
0927 16:41:29.623362 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:29.623411 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":116,"threads_started":1}
I20260927 16:41:29.791393 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:29.790954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:48205)
0927 16:41:29.791193 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:29.791295 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":65,"threads_started":1}
I20260927 16:41:30.526196 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:30.525857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:42855)
0927 16:41:30.526062 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:30.526107 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":63,"threads_started":1}
I20260927 16:41:30.624419 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:30.624077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:51323)
0927 16:41:30.624280 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:30.624340 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":63,"threads_started":1}
I20260927 16:41:30.792407 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:30.792051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:59221)
0927 16:41:30.792265 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:30.792321 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":67,"threads_started":1}
I20260927 16:41:31.398557 15752 raft_consensus.cc:482] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:41:31.398677 15752 raft_consensus.cc:504] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } }
I20260927 16:41:31.399107 15752 leader_election.cc:288] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3b5a90dc9f9d41de9b4fabb7c5e21be8 (127.0.98.3:39537), 48fe5a20c28142bca8ee7a378df77b86 (127.0.98.2:36877)
W20260927 16:41:31.399868 15339 leader_election.cc:341] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48fe5a20c28142bca8ee7a378df77b86 (127.0.98.2:36877): Not found: Tablet not found: 3cb83a7c26aa49a6a7aea5fc0222871f
W20260927 16:41:31.399950 15341 leader_election.cc:341] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3b5a90dc9f9d41de9b4fabb7c5e21be8 (127.0.98.3:39537): Not found: Tablet not found: 3cb83a7c26aa49a6a7aea5fc0222871f
I20260927 16:41:31.400019 15341 leader_election.cc:302] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbff7cfa13fc4c1ab07b892db9b7f105; no voters: 3b5a90dc9f9d41de9b4fabb7c5e21be8, 48fe5a20c28142bca8ee7a378df77b86
I20260927 16:41:31.400172 15752 raft_consensus.cc:2726] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:41:31.527288 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:31.526869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:43415)
0927 16:41:31.527086 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:31.527137 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":66,"threads_started":1}
I20260927 16:41:31.625451 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:31.625093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:39821)
0927 16:41:31.625323 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:31.625370 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":68,"threads_started":1}
I20260927 16:41:31.793375 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:31.793011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:59467)
0927 16:41:31.793247 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:31.793296 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":71,"threads_started":1}
I20260927 16:41:32.528512 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:32.528121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:59989)
0927 16:41:32.528347 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:32.528411 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":66,"threads_started":1}
I20260927 16:41:32.626443 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:32.626117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:38619)
0927 16:41:32.626323 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:32.626367 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":73,"threads_started":1}
I20260927 16:41:32.794377 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:32.794037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:46913)
0927 16:41:32.794253 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:32.794301 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":64,"threads_started":1}
I20260927 16:41:33.529587 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:33.529189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:34173)
0927 16:41:33.529446 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:33.529496 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":87,"threads_started":1}
I20260927 16:41:33.627504 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:33.627095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:52247)
0927 16:41:33.627372 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:33.627422 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":77,"threads_started":1}
I20260927 16:41:33.709677 15761 raft_consensus.cc:482] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:41:33.709823 15761 raft_consensus.cc:504] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5a90dc9f9d41de9b4fabb7c5e21be8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "48fe5a20c28142bca8ee7a378df77b86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } } peers { permanent_uuid: "bbff7cfa13fc4c1ab07b892db9b7f105" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33767 } }
I20260927 16:41:33.710291 15761 leader_election.cc:288] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3b5a90dc9f9d41de9b4fabb7c5e21be8 (127.0.98.3:39537), 48fe5a20c28142bca8ee7a378df77b86 (127.0.98.2:36877)
W20260927 16:41:33.710990 15341 leader_election.cc:341] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3b5a90dc9f9d41de9b4fabb7c5e21be8 (127.0.98.3:39537): Not found: Tablet not found: 3cb83a7c26aa49a6a7aea5fc0222871f
W20260927 16:41:33.711097 15339 leader_election.cc:341] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48fe5a20c28142bca8ee7a378df77b86 (127.0.98.2:36877): Not found: Tablet not found: 3cb83a7c26aa49a6a7aea5fc0222871f
I20260927 16:41:33.711236 15339 leader_election.cc:302] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbff7cfa13fc4c1ab07b892db9b7f105; no voters: 3b5a90dc9f9d41de9b4fabb7c5e21be8, 48fe5a20c28142bca8ee7a378df77b86
I20260927 16:41:33.711426 15761 raft_consensus.cc:2726] T 3cb83a7c26aa49a6a7aea5fc0222871f P bbff7cfa13fc4c1ab07b892db9b7f105 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:41:33.795537 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:33.795109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:35477)
0927 16:41:33.795403 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:33.795459 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":120,"threads_started":1}
I20260927 16:41:34.530577 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:34.530231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:38629)
0927 16:41:34.530443 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:34.530490 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":65,"threads_started":1}
I20260927 16:41:34.628515 15764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:34.628129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:51949)
0927 16:41:34.628360 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:34.628421 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":65,"threads_started":1}
I20260927 16:41:34.796522 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:34.796154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:42995)
0927 16:41:34.796392 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:34.796438 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":97,"threads_started":1}
I20260927 16:41:35.531749 15766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:35.531304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:56097)
0927 16:41:35.531595 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:35.531658 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":87,"threads_started":1}
I20260927 16:41:35.629470 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:35.629108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:48199)
0927 16:41:35.629335 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:35.629384 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":78,"threads_started":1}
I20260927 16:41:35.797638 15768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:35.797267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:43393)
0927 16:41:35.797500 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:35.797560 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":65,"threads_started":1}
I20260927 16:41:36.532855 15769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:36.532487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:41173)
0927 16:41:36.532710 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:36.532759 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":69,"threads_started":1}
I20260927 16:41:36.630491 15770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:36.630085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:44541)
0927 16:41:36.630348 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:36.630406 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":67,"threads_started":1}
I20260927 16:41:36.798755 15771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:36.798258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.3:35549)
0927 16:41:36.798621 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:36.798672 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":104,"threads_started":1}
I20260927 16:41:37.534119 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:37.533724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.1:60831)
0927 16:41:37.533977 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:37.534027 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":80,"threads_started":1}
I20260927 16:41:37.631572 15773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:37.631197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45119 (local address 127.0.98.2:45715)
0927 16:41:37.631427 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:37.631476 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":62,"threads_started":1}
I20260927 16:41:37.783079 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 15288
I20260927 16:41:37.788769 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 15454
I20260927 16:41:37.793898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 15589
2026-09-27T16:41:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18801 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:41:37 dist-test-slave-z40b krb5kdc[15776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:41:37 dist-test-slave-z40b krb5kdc[15776](info): set up 2 sockets
Sep 27 16:41:37 dist-test-slave-z40b krb5kdc[15776](info): commencing operation
krb5kdc: starting...
W20260927 16:41:38.824275 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:41:38 dist-test-slave-z40b krb5kdc[15776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:41:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:41:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:41:38.938138 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34029
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:41:39.039165 15789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:39.039390 15789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:39.039443 15789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:39.043258 15789 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 16:41:39.043299 15789 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 16:41:39.043320 15789 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 16:41:39.043334 15789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:41:39.043392 15789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:39.043428 15789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:39.043454 15789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:41:39.043512 15789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:41:39.048429 15789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:39.049368 15789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:39.050324 15789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:41:39.058108 15795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:39.058128 15797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:39.058144 15794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:39.058265 15789 server_base.cc:1034] running on GCE node
I20260927 16:41:39.058827 15789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:39.059808 15789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:39.060992 15789 hybrid_clock.cc:648] HybridClock initialized: now 1790527299060963 us; error 50 us; skew 500 ppm
Sep 27 16:41:39 dist-test-slave-z40b krb5kdc[15776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:39.063561 15789 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:41:39.064318 15789 webserver.cc:460] Webserver started at http://127.0.98.62:32829/ using document root <none> and password file <none>
I20260927 16:41:39.064772 15789 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:39.064837 15789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:41:39.064985 15789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:41:39.066738 15789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "3efdfc7c9e424048a60b502265e87735"
format_stamp: "Formatted at 2026-09-27 16:41:39 on dist-test-slave-z40b"
server_key: "0ae97af13245a9eb3269e28140fdbbf1"
server_key_iv: "304733397a6a58c78a2b838dba9acf39"
server_key_version: "encryptionkey@0"
I20260927 16:41:39.067301 15789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "3efdfc7c9e424048a60b502265e87735"
format_stamp: "Formatted at 2026-09-27 16:41:39 on dist-test-slave-z40b"
server_key: "0ae97af13245a9eb3269e28140fdbbf1"
server_key_iv: "304733397a6a58c78a2b838dba9acf39"
server_key_version: "encryptionkey@0"
I20260927 16:41:39.070580 15789 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260927 16:41:39.072777 15804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:39.073998 15789 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:41:39.074122 15789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "3efdfc7c9e424048a60b502265e87735"
format_stamp: "Formatted at 2026-09-27 16:41:39 on dist-test-slave-z40b"
server_key: "0ae97af13245a9eb3269e28140fdbbf1"
server_key_iv: "304733397a6a58c78a2b838dba9acf39"
server_key_version: "encryptionkey@0"
I20260927 16:41:39.074221 15789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:39.093415 15789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:39.095947 15789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:39.096139 15789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:39.102463 15789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34029
I20260927 16:41:39.102741 15856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34029 every 8 connection(s)
I20260927 16:41:39.103377 15789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:41:39.105831 15857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:41:39.110821 15857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735: Bootstrap starting.
I20260927 16:41:39.113099 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 15789
I20260927 16:41:39.113200 15857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735: Neither blocks nor log segments found. Creating new log.
I20260927 16:41:39.113229 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:41:39.113577 392 external_mini_cluster.cc:1428] Setting key 20c350db186f83c11843c8ab6ad791db
I20260927 16:41:39.114049 15857 log.cc:824] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735: Log is configured to *not* fsync() on all Append() calls
I20260927 16:41:39.115904 15857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735: No bootstrap required, opened a new log
I20260927 16:41:39.118888 15857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efdfc7c9e424048a60b502265e87735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } }
I20260927 16:41:39.119112 15857 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:41:39.119185 15857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdfc7c9e424048a60b502265e87735, State: Initialized, Role: FOLLOWER
I20260927 16:41:39.119645 15857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efdfc7c9e424048a60b502265e87735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } }
I20260927 16:41:39.119783 15857 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:41:39.119871 15857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:41:39.119966 15857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:41:39.120770 15857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efdfc7c9e424048a60b502265e87735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } }
I20260927 16:41:39.121105 15857 leader_election.cc:302] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3efdfc7c9e424048a60b502265e87735; no voters:
I20260927 16:41:39.121385 15857 leader_election.cc:288] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [CANDIDATE]: Term 1 election: Requested vote from peers
Sep 27 16:41:39 dist-test-slave-z40b krb5kdc[15776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:41:39.121487 15862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:41:39.121690 15862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [term 1 LEADER]: Becoming Leader. State: Replica: 3efdfc7c9e424048a60b502265e87735, State: Running, Role: LEADER
I20260927 16:41:39.122056 15862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efdfc7c9e424048a60b502265e87735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } }
I20260927 16:41:39.122611 15857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:41:39.123716 15864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3efdfc7c9e424048a60b502265e87735. Latest consensus state: current_term: 1 leader_uuid: "3efdfc7c9e424048a60b502265e87735" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efdfc7c9e424048a60b502265e87735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } } }
I20260927 16:41:39.123826 15864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [sys.catalog]: This master's current role is: LEADER
I20260927 16:41:39.123993 15863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3efdfc7c9e424048a60b502265e87735" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efdfc7c9e424048a60b502265e87735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } } }
I20260927 16:41:39.124742 15863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [sys.catalog]: This master's current role is: LEADER
I20260927 16:41:39.125200 15877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:41:39.128079 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:39.114601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58856 (local address 127.0.98.62:34029)
0927 16:41:39.115021 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:39.115039 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:39.115068 (+ 29us) server_negotiation.cc:407] Connection header received
0927 16:41:39.115728 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:39.115744 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:39.116064 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:39.116355 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:39.117061 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.117961 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.118764 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.119343 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.119509 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:39.122069 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:39.122075 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:39.122107 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:39.122135 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:39.125155 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:39.125658 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:39.125664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:39.125669 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:39.125782 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:39.126055 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:39.126058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:39.126060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:39.126497 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:39.126656 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:39.126922 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 16:41:39.127169 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":113,"threads_started":1}
I20260927 16:41:39.128321 15877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:41:39.132965 15877 catalog_manager.cc:1347] Generated new cluster ID: 843cf9e83d594d3fbdce518e6f4e9f17
I20260927 16:41:39.133019 15877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:41:39.159065 15877 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:41:39.160403 15877 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:41:39.167652 15877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735: Generated new TSK 0
I20260927 16:41:39.168434 15877 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:41:39.196456 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34029
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:41:39.301745 15883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:39.302004 15883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:39.302080 15883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:39.306075 15883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:39.306172 15883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:39.306259 15883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:41:39.311177 15883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:39.312232 15883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:39.313213 15883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:41:39.322448 15891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:39.322448 15888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:39.322566 15889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:39.322932 15883 server_base.cc:1034] running on GCE node
I20260927 16:41:39.323417 15883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:39.324127 15883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:39.325282 15883 hybrid_clock.cc:648] HybridClock initialized: now 1790527299325268 us; error 26 us; skew 500 ppm
Sep 27 16:41:39 dist-test-slave-z40b krb5kdc[15776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:39.327898 15883 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:41:39.328691 15883 webserver.cc:460] Webserver started at http://127.0.98.1:43089/ using document root <none> and password file <none>
I20260927 16:41:39.329211 15883 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:39.329295 15883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:41:39.329509 15883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:41:39.331461 15883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/data/instance:
uuid: "de559e6b0a2c43c3a258e764ca9b884e"
format_stamp: "Formatted at 2026-09-27 16:41:39 on dist-test-slave-z40b"
server_key: "03491d74f493144878b5f3a197841d2c"
server_key_iv: "7d133fc6fe680e75dc3838c469ee0ab2"
server_key_version: "encryptionkey@0"
I20260927 16:41:39.332016 15883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/wal/instance:
uuid: "de559e6b0a2c43c3a258e764ca9b884e"
format_stamp: "Formatted at 2026-09-27 16:41:39 on dist-test-slave-z40b"
server_key: "03491d74f493144878b5f3a197841d2c"
server_key_iv: "7d133fc6fe680e75dc3838c469ee0ab2"
server_key_version: "encryptionkey@0"
I20260927 16:41:39.335506 15883 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:41:39.337666 15898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:39.338719 15883 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:41:39.338845 15883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "de559e6b0a2c43c3a258e764ca9b884e"
format_stamp: "Formatted at 2026-09-27 16:41:39 on dist-test-slave-z40b"
server_key: "03491d74f493144878b5f3a197841d2c"
server_key_iv: "7d133fc6fe680e75dc3838c469ee0ab2"
server_key_version: "encryptionkey@0"
I20260927 16:41:39.338945 15883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:39.373536 15883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:39.376446 15883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:39.376639 15883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:39.377259 15883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:41:39.378083 15883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:41:39.378158 15883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:39.378214 15883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:41:39.378235 15883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:39.386552 15883 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41713
I20260927 16:41:39.386593 16011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41713 every 8 connection(s)
I20260927 16:41:39.387462 15883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:41:39.392409 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 15883
I20260927 16:41:39.392563 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:41:39.392867 392 external_mini_cluster.cc:1428] Setting key 2963375edeb93e62529fd98bbdae3706
Sep 27 16:41:39 dist-test-slave-z40b krb5kdc[15776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:41:39.400764 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:39.389130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53015 (local address 127.0.98.62:34029)
0927 16:41:39.389259 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:39.389269 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:39.390246 (+ 977us) server_negotiation.cc:407] Connection header received
0927 16:41:39.391108 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:39.391111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:39.391165 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:39.391276 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:39.392461 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.393150 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.394458 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.394676 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.394781 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:39.397089 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:39.397095 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:39.397101 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:39.397132 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:39.398654 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:39.399310 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:39.399313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:39.399314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:39.399383 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:39.399774 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:39.399777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:39.399778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:39.399928 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:39.400000 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:39.400568 (+ 568us) server_negotiation.cc:299] Negotiation successful
0927 16:41:39.400656 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:41:39.401831 16014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:39.389481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.1:53015)
0927 16:41:39.390104 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:39.390142 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:39.390920 (+ 778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:39.391444 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:39.391450 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:39.391757 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:41:39.392286 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:39.392318 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.393282 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.393285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:39.394331 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:39.394347 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.394819 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.394822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:39.394979 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:39.395545 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:41:39.395560 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:41:39.396937 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:41:39.398771 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:39.398778 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:39.398793 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:39.399169 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:39.399478 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:39.399484 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:39.399488 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:39.399678 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:39.400033 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:41:39.400041 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:41:39.400321 (+ 280us) client_negotiation.cc:769] Sending connection context
0927 16:41:39.400539 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 16:41:39.400864 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":125,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:41:39.402961 16012 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34029
I20260927 16:41:39.403155 16012 heartbeater.cc:464] Registering TS with master...
I20260927 16:41:39.403640 16012 heartbeater.cc:511] Master 127.0.98.62:34029 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 16:41:39.405216 15821 ts_manager.cc:194] Registered new tserver with Master: de559e6b0a2c43c3a258e764ca9b884e (127.0.98.1:41713)
I20260927 16:41:39.406744 15821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:41:39.420184 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34029
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:41:39.521548 16017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:39.521763 16017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:39.521826 16017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:39.525878 16017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:39.525924 16017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:39.526021 16017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:41:39.530783 16017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:39.531699 16017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:39.532665 16017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:41:39.541623 16023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:39.541618 16022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:39.541730 16017 server_base.cc:1034] running on GCE node
W20260927 16:41:39.541615 16025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:39.542209 16017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:39.542816 16017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:39.543972 16017 hybrid_clock.cc:648] HybridClock initialized: now 1790527299543958 us; error 26 us; skew 500 ppm
Sep 27 16:41:39 dist-test-slave-z40b krb5kdc[15776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:39.546448 16017 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:41:39.547197 16017 webserver.cc:460] Webserver started at http://127.0.98.2:34781/ using document root <none> and password file <none>
I20260927 16:41:39.547777 16017 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:39.547833 16017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:41:39.547982 16017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:41:39.549738 16017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/data/instance:
uuid: "5f2955b684a2440f94a3d0223eb40dca"
format_stamp: "Formatted at 2026-09-27 16:41:39 on dist-test-slave-z40b"
server_key: "349622560a4f0dd371097cac7a4a7f74"
server_key_iv: "c29871915d99e1628ba4d7409b8d6f11"
server_key_version: "encryptionkey@0"
I20260927 16:41:39.550191 16017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5f2955b684a2440f94a3d0223eb40dca"
format_stamp: "Formatted at 2026-09-27 16:41:39 on dist-test-slave-z40b"
server_key: "349622560a4f0dd371097cac7a4a7f74"
server_key_iv: "c29871915d99e1628ba4d7409b8d6f11"
server_key_version: "encryptionkey@0"
I20260927 16:41:39.553565 16017 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:41:39.555722 16032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:39.556846 16017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:41:39.556955 16017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "5f2955b684a2440f94a3d0223eb40dca"
format_stamp: "Formatted at 2026-09-27 16:41:39 on dist-test-slave-z40b"
server_key: "349622560a4f0dd371097cac7a4a7f74"
server_key_iv: "c29871915d99e1628ba4d7409b8d6f11"
server_key_version: "encryptionkey@0"
I20260927 16:41:39.557049 16017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:39.573571 16017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:39.576560 16017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:39.576762 16017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:39.577330 16017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:41:39.578256 16017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:41:39.578320 16017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:39.578364 16017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:41:39.578387 16017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:39.586771 16017 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34077
I20260927 16:41:39.586848 16145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34077 every 8 connection(s)
I20260927 16:41:39.587706 16017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:41:39.595446 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 16017
I20260927 16:41:39.595583 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:41:39.595896 392 external_mini_cluster.cc:1428] Setting key 1ebc087c206527f95b2356865060555e
Sep 27 16:41:39 dist-test-slave-z40b krb5kdc[15776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:41:39.600880 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:39.589389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59517 (local address 127.0.98.62:34029)
0927 16:41:39.589516 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:39.589520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:39.590316 (+ 796us) server_negotiation.cc:407] Connection header received
0927 16:41:39.591566 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:39.591569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:39.591613 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:39.591685 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:39.592754 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.593268 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.594345 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.594555 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.594627 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:39.597362 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:39.597364 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:39.597366 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:39.597386 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:39.598913 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:39.599438 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:39.599443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:39.599446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:39.599536 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:39.599850 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:39.599853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:39.599854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:39.600049 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:39.600123 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:39.600570 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 16:41:39.600711 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:41:39.601662 16148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:39.589756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.2:59517)
0927 16:41:39.590164 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:39.590204 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:39.591379 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:39.591833 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:39.591839 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:39.592189 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:41:39.592630 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:39.592642 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.593380 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.593382 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:39.594238 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:39.594247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.594674 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.594677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:39.594808 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:39.595452 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:41:39.595475 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:41:39.597185 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:41:39.599027 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:39.599031 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:39.599041 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:39.599347 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:39.599623 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:39.599625 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:39.599627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:39.599746 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:39.600132 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:41:39.600136 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:41:39.600333 (+ 197us) client_negotiation.cc:769] Sending connection context
0927 16:41:39.600511 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 16:41:39.600827 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":107,"threads_started":1}
I20260927 16:41:39.602871 16146 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34029
I20260927 16:41:39.603089 16146 heartbeater.cc:464] Registering TS with master...
I20260927 16:41:39.603560 16146 heartbeater.cc:511] Master 127.0.98.62:34029 requested a full tablet report, sending...
I20260927 16:41:39.604542 15821 ts_manager.cc:194] Registered new tserver with Master: 5f2955b684a2440f94a3d0223eb40dca (127.0.98.2:34077)
I20260927 16:41:39.605273 15821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59517
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:41:39.621577 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34029
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:41:39.727589 16151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:41:39.727829 16151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:41:39.727871 16151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:41:39.731695 16151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:41:39.731740 16151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:41:39.731793 16151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:41:39.736743 16151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:41:39.737636 16151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:41:39.738696 16151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:41:39.747537 16157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:39.747547 16159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:41:39.747547 16156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:41:39.747609 16151 server_base.cc:1034] running on GCE node
I20260927 16:41:39.748205 16151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:41:39.748750 16151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:41:39.749909 16151 hybrid_clock.cc:648] HybridClock initialized: now 1790527299749883 us; error 40 us; skew 500 ppm
Sep 27 16:41:39 dist-test-slave-z40b krb5kdc[15776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:41:39.752521 16151 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:41:39.753274 16151 webserver.cc:460] Webserver started at http://127.0.98.3:40561/ using document root <none> and password file <none>
I20260927 16:41:39.753803 16151 fs_manager.cc:359] Metadata directory not provided
I20260927 16:41:39.753890 16151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:41:39.754094 16151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:41:39.755927 16151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/data/instance:
uuid: "d24b3090424a46cea6d684dd7f564ab8"
format_stamp: "Formatted at 2026-09-27 16:41:39 on dist-test-slave-z40b"
server_key: "41a6723c1d6104735193e1463a766841"
server_key_iv: "c4e523161339659862b8fb02485f173a"
server_key_version: "encryptionkey@0"
I20260927 16:41:39.756448 16151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d24b3090424a46cea6d684dd7f564ab8"
format_stamp: "Formatted at 2026-09-27 16:41:39 on dist-test-slave-z40b"
server_key: "41a6723c1d6104735193e1463a766841"
server_key_iv: "c4e523161339659862b8fb02485f173a"
server_key_version: "encryptionkey@0"
I20260927 16:41:39.759815 16151 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:41:39.762014 16166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:39.763140 16151 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:41:39.763290 16151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "d24b3090424a46cea6d684dd7f564ab8"
format_stamp: "Formatted at 2026-09-27 16:41:39 on dist-test-slave-z40b"
server_key: "41a6723c1d6104735193e1463a766841"
server_key_iv: "c4e523161339659862b8fb02485f173a"
server_key_version: "encryptionkey@0"
I20260927 16:41:39.763423 16151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:41:39.781543 16151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:41:39.784066 16151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:41:39.784265 16151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:41:39.784787 16151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:41:39.785486 16151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:41:39.785550 16151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:39.785605 16151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:41:39.785643 16151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:41:39.793571 16151 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41487
I20260927 16:41:39.793591 16279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41487 every 8 connection(s)
I20260927 16:41:39.794294 16151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:41:39.797326 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 16151
I20260927 16:41:39.797402 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:41:39.797659 392 external_mini_cluster.cc:1428] Setting key 6b8c5816374b2e597bb9cb6c105c426b
Sep 27 16:41:39 dist-test-slave-z40b krb5kdc[15776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:41:39.806133 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:39.796025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50283 (local address 127.0.98.62:34029)
0927 16:41:39.796155 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:39.796158 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:39.797051 (+ 893us) server_negotiation.cc:407] Connection header received
0927 16:41:39.797975 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:39.797978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:39.798034 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:39.798134 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:39.799321 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.799889 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.800890 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.801086 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.801158 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:39.803321 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:39.803324 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:39.803326 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:39.803347 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:39.804498 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:39.804942 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:39.804944 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:39.804945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:39.805011 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:39.805276 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:39.805278 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:39.805279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:39.805434 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:39.805504 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:39.805908 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 16:41:39.805992 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:41:39.806864 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:39.796334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.3:50283)
0927 16:41:39.796925 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:39.796961 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:39.797778 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:39.798299 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:39.798308 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:39.798652 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:41:39.799177 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:39.799194 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.799992 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.799994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:39.800790 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:39.800799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.801200 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.801202 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:39.801324 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:39.801766 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:41:39.801780 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:41:39.803112 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:41:39.804597 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:39.804601 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:39.804611 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:39.804861 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:39.805085 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:41:39.805087 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:41:39.805088 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:41:39.805202 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:41:39.805516 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:41:39.805520 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:41:39.805706 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 16:41:39.805869 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 16:41:39.806106 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":107,"threads_started":1}
I20260927 16:41:39.807951 16280 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34029
I20260927 16:41:39.808154 16280 heartbeater.cc:464] Registering TS with master...
I20260927 16:41:39.808600 16280 heartbeater.cc:511] Master 127.0.98.62:34029 requested a full tablet report, sending...
I20260927 16:41:39.809515 15821 ts_manager.cc:194] Registered new tserver with Master: d24b3090424a46cea6d684dd7f564ab8 (127.0.98.3:41487)
I20260927 16:41:39.810196 15821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50283
I20260927 16:41:39.811280 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:41:39.819693 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:39.813302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58866 (local address 127.0.98.62:34029)
0927 16:41:39.813422 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:39.813427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:39.813576 (+ 149us) server_negotiation.cc:407] Connection header received
0927 16:41:39.813652 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:39.813654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:39.813687 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:39.813747 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:41:39.814368 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.814834 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.815683 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.815876 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.815947 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:39.817082 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:41:39.817085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:41:39.817087 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:41:39.817105 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:41:39.818252 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:39.818620 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:39.818623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:39.818624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:39.818688 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:41:39.818902 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:41:39.818904 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:41:39.818905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:41:39.819073 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:41:39.819177 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:39.819430 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 16:41:39.819508 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:41:39.822180 15821 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:41:39.825381 15821 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:41:39.843339 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:39.837441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34077 (local address 127.0.0.1:35422)
0927 16:41:39.837730 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:39.837772 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:39.837902 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:39.839405 (+ 1503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:39.839408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:39.839422 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:41:39.839506 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:39.839513 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.840791 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.840795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:39.841993 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:39.842006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.842771 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.842775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:39.842888 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:39.842926 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 16:41:39.843027 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 16:41:39.843158 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":68,"spinlock_wait_cycles":2304,"thread_start_us":87,"threads_started":1}
I20260927 16:41:39.843377 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:39.838094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35422 (local address 127.0.98.2:34077)
0927 16:41:39.838956 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:39.838965 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:39.838982 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:41:39.839039 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:39.839043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:39.839265 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:39.839427 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:41:39.839598 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.840646 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.842100 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.842647 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.842770 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:39.843066 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:39.843121 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 16:41:39.843233 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":96,"threads_started":1}
I20260927 16:41:39.843655 16296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:39.838469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58512 (local address 127.0.98.1:41713)
0927 16:41:39.838826 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:39.838842 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:39.838860 (+ 18us) server_negotiation.cc:407] Connection header received
0927 16:41:39.838921 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:39.838925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:39.839063 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:39.839183 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:41:39.839631 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.840275 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.841912 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.842720 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.842851 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:39.842992 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:39.843412 (+ 420us) server_negotiation.cc:299] Negotiation successful
0927 16:41:39.843504 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":111,"threads_started":1}
I20260927 16:41:39.843703 16293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:39.838101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41487 (local address 127.0.0.1:45842)
0927 16:41:39.838299 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:39.838311 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:39.838391 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:39.839158 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:39.839164 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:39.839186 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:41:39.839344 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:39.839353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.840744 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.840747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:39.841916 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:39.841924 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.842900 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.842903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:39.842994 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:39.843025 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 16:41:39.843074 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 16:41:39.843155 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":7,"thread_start_us":84,"threads_started":1}
I20260927 16:41:39.844002 16292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:39.837511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41713 (local address 127.0.0.1:58512)
0927 16:41:39.837769 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:39.837782 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:39.837902 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:39.839274 (+ 1372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:39.839277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:39.839293 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:41:39.839403 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:39.839409 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.840433 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.840438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:39.841762 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:39.841772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.843048 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.843054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:39.843137 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:39.843155 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 16:41:39.843251 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 16:41:39.843309 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":3,"spinlock_wait_cycles":8832,"thread_start_us":90,"threads_started":1}
I20260927 16:41:39.844623 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:39.838379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.0.98.3:41487)
0927 16:41:39.838683 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:39.838694 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:39.838716 (+ 22us) server_negotiation.cc:407] Connection header received
0927 16:41:39.838787 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:39.838792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:39.839001 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:39.839157 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:41:39.839475 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.840611 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.842033 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.842774 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.843269 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:39.843421 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:39.844375 (+ 954us) server_negotiation.cc:299] Negotiation successful
0927 16:41:39.844464 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":101,"threads_started":1}
I20260927 16:41:39.846284 16080 tablet_service.cc:1467] Processing CreateTablet for tablet 4d4cb2a0ffec4f06b301056a72751603 (DEFAULT_TABLE table=test-table [id=b6bc09b51a0c4b7197a11063928b17ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:41:39.847187 16080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d4cb2a0ffec4f06b301056a72751603. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:41:39.846917 16214 tablet_service.cc:1467] Processing CreateTablet for tablet 4d4cb2a0ffec4f06b301056a72751603 (DEFAULT_TABLE table=test-table [id=b6bc09b51a0c4b7197a11063928b17ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:41:39.847962 16214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d4cb2a0ffec4f06b301056a72751603. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:41:39.848587 15946 tablet_service.cc:1467] Processing CreateTablet for tablet 4d4cb2a0ffec4f06b301056a72751603 (DEFAULT_TABLE table=test-table [id=b6bc09b51a0c4b7197a11063928b17ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:41:39.849470 15946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d4cb2a0ffec4f06b301056a72751603. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:41:39.853302 16297 tablet_bootstrap.cc:492] T 4d4cb2a0ffec4f06b301056a72751603 P 5f2955b684a2440f94a3d0223eb40dca: Bootstrap starting.
I20260927 16:41:39.854816 16298 tablet_bootstrap.cc:492] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8: Bootstrap starting.
I20260927 16:41:39.855865 16297 tablet_bootstrap.cc:654] T 4d4cb2a0ffec4f06b301056a72751603 P 5f2955b684a2440f94a3d0223eb40dca: Neither blocks nor log segments found. Creating new log.
I20260927 16:41:39.856112 16299 tablet_bootstrap.cc:492] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e: Bootstrap starting.
I20260927 16:41:39.856954 16298 tablet_bootstrap.cc:654] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8: Neither blocks nor log segments found. Creating new log.
I20260927 16:41:39.857548 16297 log.cc:824] T 4d4cb2a0ffec4f06b301056a72751603 P 5f2955b684a2440f94a3d0223eb40dca: Log is configured to *not* fsync() on all Append() calls
I20260927 16:41:39.857834 16299 tablet_bootstrap.cc:654] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e: Neither blocks nor log segments found. Creating new log.
I20260927 16:41:39.858093 16298 log.cc:824] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8: Log is configured to *not* fsync() on all Append() calls
I20260927 16:41:39.858899 16299 log.cc:824] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e: Log is configured to *not* fsync() on all Append() calls
I20260927 16:41:39.859391 16297 tablet_bootstrap.cc:492] T 4d4cb2a0ffec4f06b301056a72751603 P 5f2955b684a2440f94a3d0223eb40dca: No bootstrap required, opened a new log
I20260927 16:41:39.859580 16297 ts_tablet_manager.cc:1397] T 4d4cb2a0ffec4f06b301056a72751603 P 5f2955b684a2440f94a3d0223eb40dca: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:41:39.859853 16298 tablet_bootstrap.cc:492] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8: No bootstrap required, opened a new log
I20260927 16:41:39.860016 16298 ts_tablet_manager.cc:1397] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:41:39.860543 16299 tablet_bootstrap.cc:492] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e: No bootstrap required, opened a new log
I20260927 16:41:39.860731 16299 ts_tablet_manager.cc:1397] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:41:39.863389 16297 raft_consensus.cc:348] T 4d4cb2a0ffec4f06b301056a72751603 P 5f2955b684a2440f94a3d0223eb40dca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } }
I20260927 16:41:39.863489 16298 raft_consensus.cc:348] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } }
I20260927 16:41:39.863657 16297 raft_consensus.cc:374] T 4d4cb2a0ffec4f06b301056a72751603 P 5f2955b684a2440f94a3d0223eb40dca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:41:39.863682 16298 raft_consensus.cc:374] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:41:39.863730 16298 raft_consensus.cc:729] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d24b3090424a46cea6d684dd7f564ab8, State: Initialized, Role: FOLLOWER
I20260927 16:41:39.863754 16297 raft_consensus.cc:729] T 4d4cb2a0ffec4f06b301056a72751603 P 5f2955b684a2440f94a3d0223eb40dca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f2955b684a2440f94a3d0223eb40dca, State: Initialized, Role: FOLLOWER
I20260927 16:41:39.863857 16299 raft_consensus.cc:348] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } }
I20260927 16:41:39.864110 16299 raft_consensus.cc:374] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:41:39.864169 16299 raft_consensus.cc:729] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de559e6b0a2c43c3a258e764ca9b884e, State: Initialized, Role: FOLLOWER
I20260927 16:41:39.864183 16298 consensus_queue.cc:260] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } }
I20260927 16:41:39.864183 16297 consensus_queue.cc:260] T 4d4cb2a0ffec4f06b301056a72751603 P 5f2955b684a2440f94a3d0223eb40dca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } }
I20260927 16:41:39.864553 16299 consensus_queue.cc:260] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } }
I20260927 16:41:39.864774 16146 heartbeater.cc:503] Master 127.0.98.62:34029 was elected leader, sending a full tablet report...
I20260927 16:41:39.865049 16297 ts_tablet_manager.cc:1428] T 4d4cb2a0ffec4f06b301056a72751603 P 5f2955b684a2440f94a3d0223eb40dca: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 16:41:39.865445 16299 ts_tablet_manager.cc:1428] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 16:41:39.865435 16280 heartbeater.cc:503] Master 127.0.98.62:34029 was elected leader, sending a full tablet report...
I20260927 16:41:39.865768 16298 ts_tablet_manager.cc:1428] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:41:39.865916 16012 heartbeater.cc:503] Master 127.0.98.62:34029 was elected leader, sending a full tablet report...
W20260927 16:41:39.888438 16013 tablet.cc:2369] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:41:39.963758 16304 raft_consensus.cc:482] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:41:39.963928 16304 raft_consensus.cc:504] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } }
I20260927 16:41:39.964998 16304 leader_election.cc:288] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f2955b684a2440f94a3d0223eb40dca (127.0.98.2:34077), de559e6b0a2c43c3a258e764ca9b884e (127.0.98.1:41713)
I20260927 16:41:39.968703 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:39.965164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34077 (local address 127.0.98.3:46915)
0927 16:41:39.965265 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:39.965282 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:39.965378 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:39.965687 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:39.965689 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:39.965706 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:41:39.965805 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:39.965815 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.966677 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.966680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:39.967619 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:39.967627 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.968269 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.968273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:39.968385 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:39.968413 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 16:41:39.968508 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 16:41:39.968579 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260927 16:41:39.968703 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:39.965255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46915 (local address 127.0.98.2:34077)
0927 16:41:39.965461 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:39.965467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:39.965483 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:41:39.965533 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:39.965536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:39.965584 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:39.965666 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:41:39.965914 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.966566 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.967739 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.968149 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.968271 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:39.968333 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:39.968537 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 16:41:39.968600 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260927 16:41:39.968890 16296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:39.965429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49239 (local address 127.0.98.1:41713)
0927 16:41:39.965553 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:39.965556 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:39.965899 (+ 343us) server_negotiation.cc:407] Connection header received
0927 16:41:39.965944 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:39.965946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:39.965984 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:39.966037 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:41:39.966288 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.966824 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.967953 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.968344 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.968461 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:39.968532 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:39.968718 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 16:41:39.968780 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:41:39.968902 16306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:39.965228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41713 (local address 127.0.98.3:49239)
0927 16:41:39.965581 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:39.965593 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:41:39.965673 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:41:39.966072 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:41:39.966075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:41:39.966093 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:41:39.966199 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:39.966205 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.966924 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.966930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:39.967844 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:41:39.967856 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.968463 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.968466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:41:39.968583 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:39.968603 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 16:41:39.968702 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 16:41:39.968765 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":150,"threads_started":1}
I20260927 16:41:39.969393 16100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d4cb2a0ffec4f06b301056a72751603" candidate_uuid: "d24b3090424a46cea6d684dd7f564ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f2955b684a2440f94a3d0223eb40dca" is_pre_election: true
I20260927 16:41:39.969532 15966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d4cb2a0ffec4f06b301056a72751603" candidate_uuid: "d24b3090424a46cea6d684dd7f564ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de559e6b0a2c43c3a258e764ca9b884e" is_pre_election: true
I20260927 16:41:39.969693 16100 raft_consensus.cc:2454] T 4d4cb2a0ffec4f06b301056a72751603 P 5f2955b684a2440f94a3d0223eb40dca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d24b3090424a46cea6d684dd7f564ab8 in term 0.
I20260927 16:41:39.969758 15966 raft_consensus.cc:2454] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d24b3090424a46cea6d684dd7f564ab8 in term 0.
I20260927 16:41:39.970167 16167 leader_election.cc:302] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d24b3090424a46cea6d684dd7f564ab8, de559e6b0a2c43c3a258e764ca9b884e; no voters:
I20260927 16:41:39.970495 16304 raft_consensus.cc:2781] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:41:39.970573 16304 raft_consensus.cc:482] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:41:39.970604 16304 raft_consensus.cc:3037] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:41:39.971649 16304 raft_consensus.cc:504] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } }
I20260927 16:41:39.972018 16304 leader_election.cc:288] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8 [CANDIDATE]: Term 1 election: Requested vote from peers 5f2955b684a2440f94a3d0223eb40dca (127.0.98.2:34077), de559e6b0a2c43c3a258e764ca9b884e (127.0.98.1:41713)
I20260927 16:41:39.972343 16100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d4cb2a0ffec4f06b301056a72751603" candidate_uuid: "d24b3090424a46cea6d684dd7f564ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f2955b684a2440f94a3d0223eb40dca"
I20260927 16:41:39.972393 15966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d4cb2a0ffec4f06b301056a72751603" candidate_uuid: "d24b3090424a46cea6d684dd7f564ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de559e6b0a2c43c3a258e764ca9b884e"
I20260927 16:41:39.972453 16100 raft_consensus.cc:3037] T 4d4cb2a0ffec4f06b301056a72751603 P 5f2955b684a2440f94a3d0223eb40dca [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:41:39.972535 15966 raft_consensus.cc:3037] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:41:39.973448 16100 raft_consensus.cc:2454] T 4d4cb2a0ffec4f06b301056a72751603 P 5f2955b684a2440f94a3d0223eb40dca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d24b3090424a46cea6d684dd7f564ab8 in term 1.
I20260927 16:41:39.973553 15966 raft_consensus.cc:2454] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d24b3090424a46cea6d684dd7f564ab8 in term 1.
I20260927 16:41:39.973754 16169 leader_election.cc:302] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f2955b684a2440f94a3d0223eb40dca, d24b3090424a46cea6d684dd7f564ab8; no voters:
I20260927 16:41:39.973898 16304 raft_consensus.cc:2781] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:41:39.974098 16304 raft_consensus.cc:686] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8 [term 1 LEADER]: Becoming Leader. State: Replica: d24b3090424a46cea6d684dd7f564ab8, State: Running, Role: LEADER
I20260927 16:41:39.974423 16304 consensus_queue.cc:237] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } }
I20260927 16:41:39.977435 15819 catalog_manager.cc:5495] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8 reported cstate change: term changed from 0 to 1, leader changed from <none> to d24b3090424a46cea6d684dd7f564ab8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d24b3090424a46cea6d684dd7f564ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } health_report { overall_health: HEALTHY } } }
I20260927 16:41:39.995713 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:39.992556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45854 (local address 127.0.98.3:41487)
0927 16:41:39.992668 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 16:41:39.992671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:41:39.992686 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:41:39.992834 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:41:39.992836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:41:39.992883 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:41:39.992965 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:41:39.993168 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.993620 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.994416 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:41:39.994605 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:41:39.994674 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:41:39.994909 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:41:39.995353 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:41:39.995438 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 16:41:39.995497 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 16:41:39.995537 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:41:40.001245 15966 raft_consensus.cc:1261] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e [term 1 FOLLOWER]: Refusing update from remote peer d24b3090424a46cea6d684dd7f564ab8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:41:40.001262 16100 raft_consensus.cc:1261] T 4d4cb2a0ffec4f06b301056a72751603 P 5f2955b684a2440f94a3d0223eb40dca [term 1 FOLLOWER]: Refusing update from remote peer d24b3090424a46cea6d684dd7f564ab8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:41:40.001950 16304 consensus_queue.cc:1035] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:41:40.002169 16307 consensus_queue.cc:1035] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:41:40.008180 16310 mvcc.cc:205] Tried to move back new op lower bound from 7333999820797878272 to 7333999820697985024. Current Snapshot: MvccSnapshot[applied={T|T < 7333999820797878272}]
I20260927 16:41:40.009753 16312 mvcc.cc:205] Tried to move back new op lower bound from 7333999820797878272 to 7333999820697985024. Current Snapshot: MvccSnapshot[applied={T|T < 7333999820797878272}]
I20260927 16:41:40.016407 15818 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:41:40.016567 15818 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:41:40.018692 15818 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735: Sending DeleteTablet for 3 replicas of tablet 4d4cb2a0ffec4f06b301056a72751603
I20260927 16:41:40.019408 16080 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d4cb2a0ffec4f06b301056a72751603 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:41:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35422
I20260927 16:41:40.019449 16214 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d4cb2a0ffec4f06b301056a72751603 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:41:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45842
I20260927 16:41:40.019838 16319 tablet_replica.cc:330] T 4d4cb2a0ffec4f06b301056a72751603 P 5f2955b684a2440f94a3d0223eb40dca: stopping tablet replica
I20260927 16:41:40.019886 15946 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d4cb2a0ffec4f06b301056a72751603 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:41:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58512
I20260927 16:41:40.020061 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 15883
I20260927 16:41:40.020180 16319 raft_consensus.cc:2229] T 4d4cb2a0ffec4f06b301056a72751603 P 5f2955b684a2440f94a3d0223eb40dca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:41:40.020774 16319 raft_consensus.cc:2258] T 4d4cb2a0ffec4f06b301056a72751603 P 5f2955b684a2440f94a3d0223eb40dca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:41:40.021986 16320 tablet_replica.cc:330] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8: stopping tablet replica
I20260927 16:41:40.022287 16320 raft_consensus.cc:2229] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:41:40.022632 16320 raft_consensus.cc:2258] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:41:40.022686 16319 ts_tablet_manager.cc:1905] T 4d4cb2a0ffec4f06b301056a72751603 P 5f2955b684a2440f94a3d0223eb40dca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:41:40.024120 16320 ts_tablet_manager.cc:1905] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:41:40.025517 16319 ts_tablet_manager.cc:1918] T 4d4cb2a0ffec4f06b301056a72751603 P 5f2955b684a2440f94a3d0223eb40dca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:41:40.025661 16319 log.cc:1196] T 4d4cb2a0ffec4f06b301056a72751603 P 5f2955b684a2440f94a3d0223eb40dca: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/wal/wals/4d4cb2a0ffec4f06b301056a72751603
I20260927 16:41:40.026039 16319 ts_tablet_manager.cc:1939] T 4d4cb2a0ffec4f06b301056a72751603 P 5f2955b684a2440f94a3d0223eb40dca: Deleting consensus metadata
I20260927 16:41:40.026960 16320 ts_tablet_manager.cc:1918] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:41:40.027113 16320 log.cc:1196] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/wal/wals/4d4cb2a0ffec4f06b301056a72751603
I20260927 16:41:40.027343 15807 catalog_manager.cc:4841] TS 5f2955b684a2440f94a3d0223eb40dca (127.0.98.2:34077): tablet 4d4cb2a0ffec4f06b301056a72751603 (table test-table [id=b6bc09b51a0c4b7197a11063928b17ab]) successfully deleted
I20260927 16:41:40.027527 16320 ts_tablet_manager.cc:1939] T 4d4cb2a0ffec4f06b301056a72751603 P d24b3090424a46cea6d684dd7f564ab8: Deleting consensus metadata
W20260927 16:41:40.027570 15805 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41713 (error 108)
I20260927 16:41:40.028540 16292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:41:40.028217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41713 (local address 127.0.0.1:58526)
0927 16:41:40.028356 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 16:41:40.028434 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 16:41:40.028568 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 16017
I20260927 16:41:40.028570 15805 catalog_manager.cc:4841] TS d24b3090424a46cea6d684dd7f564ab8 (127.0.98.3:41487): tablet 4d4cb2a0ffec4f06b301056a72751603 (table test-table [id=b6bc09b51a0c4b7197a11063928b17ab]) successfully deleted
W20260927 16:41:40.028709 15805 catalog_manager.cc:4584] TS de559e6b0a2c43c3a258e764ca9b884e (127.0.98.1:41713): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4d4cb2a0ffec4f06b301056a72751603: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41713: connect: Connection refused (error 111)
I20260927 16:41:40.034343 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 16151
I20260927 16:41:40.038781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 15789
I20260927 16:42:00.044714 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34029
--webserver_interface=127.0.98.62
--webserver_port=32829
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:42:00.148725 16323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:00.148929 16323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:00.148963 16323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:00.152930 16323 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 16:42:00.152973 16323 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 16:42:00.152998 16323 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 16:42:00.153019 16323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:42:00.153069 16323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:00.153105 16323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:00.153136 16323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:42:00.153208 16323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:42:00.158147 16323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=32829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:00.159061 16323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:00.160006 16323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:00.172153 16329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:00.172261 16323 server_base.cc:1034] running on GCE node
W20260927 16:42:00.172171 16332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:00.172158 16330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:00.172976 16323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:00.174005 16323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:00.175177 16323 hybrid_clock.cc:648] HybridClock initialized: now 1790527320175147 us; error 44 us; skew 500 ppm
Sep 27 16:42:00 dist-test-slave-z40b krb5kdc[15776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:00.177677 16323 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:42:00.178454 16323 webserver.cc:460] Webserver started at http://127.0.98.62:32829/ using document root <none> and password file <none>
I20260927 16:42:00.178983 16323 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:00.179076 16323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:00.182639 16323 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:42:00.184648 16339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:00.185729 16323 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 16:42:00.185855 16323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "3efdfc7c9e424048a60b502265e87735"
format_stamp: "Formatted at 2026-09-27 16:41:39 on dist-test-slave-z40b"
server_key: "0ae97af13245a9eb3269e28140fdbbf1"
server_key_iv: "304733397a6a58c78a2b838dba9acf39"
server_key_version: "encryptionkey@0"
I20260927 16:42:00.186388 16323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:00.204833 16323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:00.207366 16323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:00.207559 16323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:00.214468 16323 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34029
I20260927 16:42:00.214726 16391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34029 every 8 connection(s)
I20260927 16:42:00.215533 16323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:42:00.219527 16392 sys_catalog.cc:263] Verifying existing consensus state
I20260927 16:42:00.220640 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 16323
I20260927 16:42:00.220949 16392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735: Bootstrap starting.
I20260927 16:42:00.221055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41713
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34029
--webserver_port=43089
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:42:00.237283 16392 log.cc:824] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:00.245209 16392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:42:00.245543 16392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735: Bootstrap complete.
I20260927 16:42:00.249442 16392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efdfc7c9e424048a60b502265e87735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } }
I20260927 16:42:00.249804 16392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdfc7c9e424048a60b502265e87735, State: Initialized, Role: FOLLOWER
I20260927 16:42:00.250322 16392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efdfc7c9e424048a60b502265e87735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } }
I20260927 16:42:00.250429 16392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:42:00.250512 16392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:42:00.250609 16392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [term 1 FOLLOWER]: Advancing to term 2
I20260927 16:42:00.252439 16392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efdfc7c9e424048a60b502265e87735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } }
I20260927 16:42:00.252749 16392 leader_election.cc:302] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3efdfc7c9e424048a60b502265e87735; no voters:
I20260927 16:42:00.253033 16392 leader_election.cc:288] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 16:42:00.253110 16396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 16:42:00.253307 16396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [term 2 LEADER]: Becoming Leader. State: Replica: 3efdfc7c9e424048a60b502265e87735, State: Running, Role: LEADER
I20260927 16:42:00.253752 16396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efdfc7c9e424048a60b502265e87735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } }
I20260927 16:42:00.254240 16392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:42:00.254611 16396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3efdfc7c9e424048a60b502265e87735. Latest consensus state: current_term: 2 leader_uuid: "3efdfc7c9e424048a60b502265e87735" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efdfc7c9e424048a60b502265e87735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } } }
I20260927 16:42:00.254793 16396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:00.255057 16397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3efdfc7c9e424048a60b502265e87735" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efdfc7c9e424048a60b502265e87735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } } }
I20260927 16:42:00.255239 16397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:00.255252 16400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:42:00.259711 16400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:42:00.260442 16400 catalog_manager.cc:1259] Loaded cluster ID: 843cf9e83d594d3fbdce518e6f4e9f17
I20260927 16:42:00.260493 16400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:42:00.262818 16400 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:42:00.263696 16400 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735: Loaded TSK: 0
I20260927 16:42:00.284478 16400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735: Generated new TSK 1
I20260927 16:42:00.284801 16400 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 16:42:00.332432 16394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:00.332643 16394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:00.332682 16394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:00.336954 16394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:00.337013 16394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:00.337127 16394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:42:00.341854 16394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41713
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43089
--tserver_master_addrs=127.0.98.62:34029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:00.342685 16394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:00.343660 16394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:00.352082 16419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:00.352188 16422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:00.352092 16420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:00.352537 16394 server_base.cc:1034] running on GCE node
I20260927 16:42:00.352957 16394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:00.353638 16394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:00.354854 16394 hybrid_clock.cc:648] HybridClock initialized: now 1790527320354821 us; error 43 us; skew 500 ppm
Sep 27 16:42:00 dist-test-slave-z40b krb5kdc[15776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:00.357947 16394 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:42:00.358768 16394 webserver.cc:460] Webserver started at http://127.0.98.1:43089/ using document root <none> and password file <none>
I20260927 16:42:00.359314 16394 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:00.359406 16394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:00.362949 16394 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:42:00.365068 16429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:00.366272 16394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 16:42:00.366390 16394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "de559e6b0a2c43c3a258e764ca9b884e"
format_stamp: "Formatted at 2026-09-27 16:41:39 on dist-test-slave-z40b"
server_key: "03491d74f493144878b5f3a197841d2c"
server_key_iv: "7d133fc6fe680e75dc3838c469ee0ab2"
server_key_version: "encryptionkey@0"
I20260927 16:42:00.366948 16394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:00.401181 16394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:00.404371 16394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:00.404561 16394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:00.405145 16394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:42:00.406191 16436 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 16:42:00.410046 16394 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 16:42:00.410127 16394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260927 16:42:00.410198 16394 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 16:42:00.411830 16394 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 16:42:00.411892 16394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260927 16:42:00.412153 16436 tablet_bootstrap.cc:492] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e: Bootstrap starting.
I20260927 16:42:00.424113 16394 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41713
I20260927 16:42:00.424185 16543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41713 every 8 connection(s)
I20260927 16:42:00.425261 16394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:42:00.427416 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 16394
I20260927 16:42:00.428026 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34077
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34029
--webserver_port=34781
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:42:00.433432 16436 log.cc:824] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e: Log is configured to *not* fsync() on all Append() calls
Sep 27 16:42:00 dist-test-slave-z40b krb5kdc[15776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:42:00.440903 16436 tablet_bootstrap.cc:492] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:42:00.441232 16436 tablet_bootstrap.cc:492] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e: Bootstrap complete.
I20260927 16:42:00.442076 16547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:00.427163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.1:54079)
0927 16:42:00.427869 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:00.427920 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:00.428723 (+ 803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:00.430095 (+ 1372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:00.430102 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:00.430496 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:42:00.431095 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:00.431112 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.432388 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.432390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:00.433153 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:00.433162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.434131 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.434135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:00.434346 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:00.435031 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:42:00.435048 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:42:00.436865 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:42:00.439138 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:00.439144 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:00.439155 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:00.439465 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:00.439810 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:00.439813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:00.439816 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:00.439959 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:00.440552 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:42:00.440558 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:42:00.440827 (+ 269us) client_negotiation.cc:769] Sending connection context
0927 16:42:00.440980 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 16:42:00.441244 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":132,"threads_started":1}
I20260927 16:42:00.442260 16436 ts_tablet_manager.cc:1397] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e: Time spent bootstrapping tablet: real 0.030s user 0.017s sys 0.008s
I20260927 16:42:00.443583 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:00.427302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54079 (local address 127.0.98.62:34029)
0927 16:42:00.429066 (+ 1764us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:00.429078 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:00.429106 (+ 28us) server_negotiation.cc:407] Connection header received
0927 16:42:00.429310 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:00.429325 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:00.429690 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:00.430017 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:00.431297 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.432237 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.433303 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.433981 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.434146 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:00.437058 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:00.437064 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:00.437076 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:00.437104 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:00.439022 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:00.439584 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:00.439589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:00.439594 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:00.439721 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:00.440056 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:00.440058 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:00.440060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:00.440447 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:00.440567 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:00.442507 (+ 1940us) server_negotiation.cc:299] Negotiation successful
0927 16:42:00.442799 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1657,"thread_start_us":113,"threads_started":1}
I20260927 16:42:00.445256 16436 raft_consensus.cc:348] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } }
I20260927 16:42:00.445652 16544 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34029
I20260927 16:42:00.445710 16436 raft_consensus.cc:729] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: de559e6b0a2c43c3a258e764ca9b884e, State: Initialized, Role: FOLLOWER
I20260927 16:42:00.445858 16544 heartbeater.cc:464] Registering TS with master...
I20260927 16:42:00.446192 16436 consensus_queue.cc:260] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } }
I20260927 16:42:00.446329 16544 heartbeater.cc:511] Master 127.0.98.62:34029 requested a full tablet report, sending...
I20260927 16:42:00.447113 16436 ts_tablet_manager.cc:1428] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 16:42:00.448841 16356 ts_manager.cc:194] Registered new tserver with Master: de559e6b0a2c43c3a258e764ca9b884e (127.0.98.1:41713)
I20260927 16:42:00.452447 16356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54079
I20260927 16:42:00.454792 16544 heartbeater.cc:503] Master 127.0.98.62:34029 was elected leader, sending a full tablet report...
Sep 27 16:42:00 dist-test-slave-z40b krb5kdc[15776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 16:42:00.462673 16552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:00.451878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46088 (local address 127.0.98.1:41713)
0927 16:42:00.452196 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:00.452206 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:00.452586 (+ 380us) server_negotiation.cc:407] Connection header received
0927 16:42:00.452818 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:00.452828 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:00.452961 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:00.453084 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:00.453853 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.454921 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.456215 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.456693 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.456801 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:00.459303 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:00.459311 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:00.459326 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:00.459344 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:00.460888 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:00.461422 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:00.461426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:00.461427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:00.461497 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:00.461822 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:00.461824 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:00.461825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:00.462029 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:00.462115 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:00.462427 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 16:42:00.462536 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":86,"threads_started":1}
I20260927 16:42:00.462682 16551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:00.451847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41713 (local address 127.0.0.1:46088)
0927 16:42:00.452430 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:00.452461 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:00.452681 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:00.453195 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:00.453201 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:00.453589 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:42:00.453705 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:00.453713 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.455045 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.455048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:00.456057 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:00.456068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.456821 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.456834 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:00.456913 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:00.457557 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:42:00.457583 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:42:00.459099 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:42:00.460997 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:00.461006 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:00.461010 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:00.461296 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:00.461609 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:00.461612 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:00.461614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:00.461733 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:00.462126 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:42:00.462131 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:42:00.462237 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 16:42:00.462368 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 16:42:00.462529 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":102,"threads_started":1}
I20260927 16:42:00.463816 16478 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d4cb2a0ffec4f06b301056a72751603 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:41:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46088
I20260927 16:42:00.464540 16553 tablet_replica.cc:330] stopping tablet replica
I20260927 16:42:00.464816 16553 raft_consensus.cc:2229] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:42:00.465032 16553 raft_consensus.cc:2258] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:42:00.465840 16553 ts_tablet_manager.cc:1905] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:42:00.469362 16553 ts_tablet_manager.cc:1918] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:42:00.469431 16553 log.cc:1196] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/wal/wals/4d4cb2a0ffec4f06b301056a72751603
I20260927 16:42:00.469708 16553 ts_tablet_manager.cc:1939] T 4d4cb2a0ffec4f06b301056a72751603 P de559e6b0a2c43c3a258e764ca9b884e: Deleting consensus metadata
I20260927 16:42:00.471304 16340 catalog_manager.cc:4841] TS de559e6b0a2c43c3a258e764ca9b884e (127.0.98.1:41713): tablet 4d4cb2a0ffec4f06b301056a72751603 (table test-table [id=b6bc09b51a0c4b7197a11063928b17ab]) successfully deleted
W20260927 16:42:00.541054 16548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:00.541260 16548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:00.541312 16548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:00.545029 16548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:00.545073 16548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:00.545151 16548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:42:00.549938 16548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34781
--tserver_master_addrs=127.0.98.62:34029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:00.550743 16548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:00.551599 16548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:00.560645 16558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:00.560705 16548 server_base.cc:1034] running on GCE node
W20260927 16:42:00.560657 16561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:00.560657 16559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:00.561157 16548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:00.561672 16548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:00.562831 16548 hybrid_clock.cc:648] HybridClock initialized: now 1790527320562820 us; error 25 us; skew 500 ppm
Sep 27 16:42:00 dist-test-slave-z40b krb5kdc[15776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:00.565364 16548 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:42:00.566090 16548 webserver.cc:460] Webserver started at http://127.0.98.2:34781/ using document root <none> and password file <none>
I20260927 16:42:00.566591 16548 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:00.566690 16548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:00.570319 16548 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:42:00.572270 16568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:00.573310 16548 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:42:00.573423 16548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "5f2955b684a2440f94a3d0223eb40dca"
format_stamp: "Formatted at 2026-09-27 16:41:39 on dist-test-slave-z40b"
server_key: "349622560a4f0dd371097cac7a4a7f74"
server_key_iv: "c29871915d99e1628ba4d7409b8d6f11"
server_key_version: "encryptionkey@0"
I20260927 16:42:00.573935 16548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:00.585628 16548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:00.588116 16548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:00.588342 16548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:00.588887 16548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:42:00.589707 16548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:42:00.589751 16548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:00.589815 16548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:42:00.589844 16548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:00.597849 16548 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34077
I20260927 16:42:00.597872 16681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34077 every 8 connection(s)
I20260927 16:42:00.598703 16548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:42:00.604779 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 16548
I20260927 16:42:00.605206 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41487
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34029
--webserver_port=40561
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 16:42:00 dist-test-slave-z40b krb5kdc[15776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:42:00.610950 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:00.600246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43397 (local address 127.0.98.62:34029)
0927 16:42:00.600369 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:00.600373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:00.601302 (+ 929us) server_negotiation.cc:407] Connection header received
0927 16:42:00.602033 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:00.602036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:00.602081 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:00.602165 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:00.603270 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.603792 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.605048 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.605286 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.605413 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:00.607623 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:00.607627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:00.607634 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:00.607659 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:00.609039 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:00.609518 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:00.609521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:00.609524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:00.609599 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:00.609910 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:00.609914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:00.609916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:00.610081 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:00.610204 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:00.610657 (+ 453us) server_negotiation.cc:299] Negotiation successful
0927 16:42:00.610783 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:42:00.611800 16684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:00.600584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.2:43397)
0927 16:42:00.601171 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:00.601231 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:00.601884 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:00.602357 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:00.602363 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:00.602685 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:42:00.603096 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:00.603108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.603898 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.603900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:00.604936 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:00.604947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.605546 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.605549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:00.605670 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:00.606150 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:42:00.606164 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:42:00.607466 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:42:00.609156 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:00.609161 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:00.609173 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:00.609414 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:00.609683 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:00.609687 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:00.609689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:00.609820 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:00.610205 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:42:00.610212 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:42:00.610432 (+ 220us) client_negotiation.cc:769] Sending connection context
0927 16:42:00.610601 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 16:42:00.610901 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":128,"threads_started":1}
I20260927 16:42:00.613036 16682 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34029
I20260927 16:42:00.613258 16682 heartbeater.cc:464] Registering TS with master...
I20260927 16:42:00.613698 16682 heartbeater.cc:511] Master 127.0.98.62:34029 requested a full tablet report, sending...
I20260927 16:42:00.614694 16356 ts_manager.cc:194] Registered new tserver with Master: 5f2955b684a2440f94a3d0223eb40dca (127.0.98.2:34077)
I20260927 16:42:00.615329 16356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43397
W20260927 16:42:00.708340 16685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:00.708572 16685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:00.708611 16685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:00.712611 16685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:00.712658 16685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:00.712746 16685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:42:00.717741 16685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40561
--tserver_master_addrs=127.0.98.62:34029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:00.718665 16685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:00.719661 16685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:00.728509 16690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:00.728485 16691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:00.728581 16685 server_base.cc:1034] running on GCE node
W20260927 16:42:00.728468 16693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:00.729027 16685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:00.729574 16685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:00.730720 16685 hybrid_clock.cc:648] HybridClock initialized: now 1790527320730687 us; error 40 us; skew 500 ppm
Sep 27 16:42:00 dist-test-slave-z40b krb5kdc[15776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:00.733527 16685 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:42:00.734319 16685 webserver.cc:460] Webserver started at http://127.0.98.3:40561/ using document root <none> and password file <none>
I20260927 16:42:00.734835 16685 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:00.734902 16685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:00.738372 16685 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:42:00.740173 16700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:00.741261 16685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:42:00.741379 16685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "d24b3090424a46cea6d684dd7f564ab8"
format_stamp: "Formatted at 2026-09-27 16:41:39 on dist-test-slave-z40b"
server_key: "41a6723c1d6104735193e1463a766841"
server_key_iv: "c4e523161339659862b8fb02485f173a"
server_key_version: "encryptionkey@0"
I20260927 16:42:00.741868 16685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:00.759512 16685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:00.762176 16685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:00.762337 16685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:00.762807 16685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:42:00.763589 16685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:42:00.763660 16685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:00.763746 16685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:42:00.763789 16685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:00.772047 16685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41487
I20260927 16:42:00.772069 16813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41487 every 8 connection(s)
I20260927 16:42:00.772889 16685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 16:42:00 dist-test-slave-z40b krb5kdc[15776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:42:00.781163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 16685
I20260927 16:42:00.785317 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:00.774716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35829 (local address 127.0.98.62:34029)
0927 16:42:00.774855 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:00.774859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:00.775754 (+ 895us) server_negotiation.cc:407] Connection header received
0927 16:42:00.776551 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:00.776553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:00.776594 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:00.776648 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:00.777669 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.778128 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.779083 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.779364 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.779435 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:00.781729 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:00.781732 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:00.781734 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:00.781763 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:00.783257 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:00.783848 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:00.783855 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:00.783860 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:00.783945 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:00.784287 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:00.784290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:00.784292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:00.784491 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:00.784585 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:00.785090 (+ 505us) server_negotiation.cc:299] Negotiation successful
0927 16:42:00.785206 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 16:42:00.786299 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:00.775054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.3:35829)
0927 16:42:00.775646 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:00.775680 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:00.776345 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:00.776783 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:00.776789 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:00.777078 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:42:00.777544 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:00.777563 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.778218 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.778220 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:00.778973 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:00.778981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.779484 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.779486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:00.779612 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:00.780022 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:42:00.780035 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:42:00.781583 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:42:00.783353 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:00.783357 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:00.783366 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:00.783734 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:00.784030 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:00.784033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:00.784035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:00.784193 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:00.784595 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:42:00.784599 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:42:00.784893 (+ 294us) client_negotiation.cc:769] Sending connection context
0927 16:42:00.785028 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 16:42:00.785280 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":146,"threads_started":1}
I20260927 16:42:00.787359 16814 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34029
I20260927 16:42:00.787578 16814 heartbeater.cc:464] Registering TS with master...
I20260927 16:42:00.788069 16814 heartbeater.cc:511] Master 127.0.98.62:34029 requested a full tablet report, sending...
I20260927 16:42:00.789075 16356 ts_manager.cc:194] Registered new tserver with Master: d24b3090424a46cea6d684dd7f564ab8 (127.0.98.3:41487)
I20260927 16:42:00.789569 16356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35829
I20260927 16:42:00.789958 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:00.782029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53526 (local address 127.0.98.62:34029)
0927 16:42:00.782297 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:00.782302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:00.782315 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:42:00.782346 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:00.782350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:00.782390 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:00.782468 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:00.783434 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.784164 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.785196 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.785527 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.785646 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:00.786799 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:00.786803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:00.786805 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:00.786826 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:00.788359 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:00.788768 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:00.788772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:00.788773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:00.788905 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:00.789206 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:00.789217 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:00.789221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:00.789374 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:00.789565 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:00.789684 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 16:42:00.789802 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":59,"threads_started":1}
I20260927 16:42:00.791564 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:42:00.795274 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:00.792092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53540 (local address 127.0.98.62:34029)
0927 16:42:00.792221 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:00.792227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:00.792351 (+ 124us) server_negotiation.cc:407] Connection header received
0927 16:42:00.792504 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:00.792507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:00.792589 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:00.792681 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:42:00.792900 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.793352 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.794165 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.794397 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.794500 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:00.794786 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:42:00.794885 (+ 99us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 16:42:00.795105 (+ 220us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53540: token expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 16:42:00.795331 16818 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53540: token expired
W20260927 16:42:00.795660 16284 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:34029)
I20260927 16:42:00.802152 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:00.795981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53550 (local address 127.0.98.62:34029)
0927 16:42:00.796075 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:00.796079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:00.796116 (+ 37us) server_negotiation.cc:407] Connection header received
0927 16:42:00.796248 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:00.796250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:00.796285 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:00.796351 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:00.796871 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.797330 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.798011 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.798214 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.798306 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:00.799504 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:00.799507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:00.799509 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:00.799534 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:00.800599 (+ 1065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:00.800984 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:00.800998 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:00.801000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:00.801076 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:00.801347 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:00.801350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:00.801352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:00.801543 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:00.801691 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:00.801909 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 16:42:00.802018 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 16:42:00.804131 16356 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:42:00.807019 16356 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:42:00.814515 16478 tablet_service.cc:1467] Processing CreateTablet for tablet bc1db73e0d8e44f880057431984bc9f3 (DEFAULT_TABLE table=test-table [id=548013ea07fa46ca9dda461daf797aea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:42:00.815150 16478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc1db73e0d8e44f880057431984bc9f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:00.817886 16436 tablet_bootstrap.cc:492] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e: Bootstrap starting.
I20260927 16:42:00.818310 16824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:00.813607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41487 (local address 127.0.0.1:46626)
0927 16:42:00.813778 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:00.813791 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:00.813873 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:00.814546 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:00.814549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:00.814601 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:00.814700 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:00.814706 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.816044 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.816053 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:00.816845 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:00.816854 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.817905 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.817909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:00.818003 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:00.818054 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 16:42:00.818114 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 16:42:00.818177 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":20,"thread_start_us":68,"threads_started":1}
I20260927 16:42:00.818444 16551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:00.813176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34077 (local address 127.0.0.1:51988)
0927 16:42:00.813297 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:00.813312 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:00.813384 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:00.814187 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:00.814190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:00.814208 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:00.814292 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:00.814298 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.815994 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.815998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:00.817117 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:00.817127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.818074 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.818078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:00.818161 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:00.818182 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 16:42:00.818239 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 16:42:00.818317 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":50}
I20260927 16:42:00.818519 16823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:00.813371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51988 (local address 127.0.98.2:34077)
0927 16:42:00.813648 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:00.813659 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:00.813682 (+ 23us) server_negotiation.cc:407] Connection header received
0927 16:42:00.813767 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:00.813772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:00.813989 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:00.814195 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:00.814390 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.815880 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.817253 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.817978 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.818074 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:00.818206 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:00.818275 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 16:42:00.818385 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":106,"threads_started":1}
I20260927 16:42:00.818694 16825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:00.813731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46626 (local address 127.0.98.3:41487)
0927 16:42:00.814102 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:00.814116 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:00.814139 (+ 23us) server_negotiation.cc:407] Connection header received
0927 16:42:00.814203 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:00.814207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:00.814402 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:00.814554 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:00.814828 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.815930 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.816967 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.817622 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.817722 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:00.817845 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:00.818468 (+ 623us) server_negotiation.cc:299] Negotiation successful
0927 16:42:00.818558 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":101,"threads_started":1}
I20260927 16:42:00.818799 16436 tablet_bootstrap.cc:654] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:00.820322 16436 tablet_bootstrap.cc:492] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e: No bootstrap required, opened a new log
I20260927 16:42:00.820408 16436 ts_tablet_manager.cc:1397] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260927 16:42:00.820794 16436 raft_consensus.cc:348] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } }
I20260927 16:42:00.820904 16436 raft_consensus.cc:374] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:00.820938 16436 raft_consensus.cc:729] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de559e6b0a2c43c3a258e764ca9b884e, State: Initialized, Role: FOLLOWER
I20260927 16:42:00.821050 16436 consensus_queue.cc:260] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } }
I20260927 16:42:00.821270 16436 ts_tablet_manager.cc:1428] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 16:42:00.821192 16616 tablet_service.cc:1467] Processing CreateTablet for tablet bc1db73e0d8e44f880057431984bc9f3 (DEFAULT_TABLE table=test-table [id=548013ea07fa46ca9dda461daf797aea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:42:00.821960 16748 tablet_service.cc:1467] Processing CreateTablet for tablet bc1db73e0d8e44f880057431984bc9f3 (DEFAULT_TABLE table=test-table [id=548013ea07fa46ca9dda461daf797aea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:42:00.822325 16616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc1db73e0d8e44f880057431984bc9f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:00.822741 16748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc1db73e0d8e44f880057431984bc9f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:00.829931 16827 tablet_bootstrap.cc:492] T bc1db73e0d8e44f880057431984bc9f3 P d24b3090424a46cea6d684dd7f564ab8: Bootstrap starting.
I20260927 16:42:00.831259 16828 tablet_bootstrap.cc:492] T bc1db73e0d8e44f880057431984bc9f3 P 5f2955b684a2440f94a3d0223eb40dca: Bootstrap starting.
I20260927 16:42:00.832206 16827 tablet_bootstrap.cc:654] T bc1db73e0d8e44f880057431984bc9f3 P d24b3090424a46cea6d684dd7f564ab8: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:00.833338 16828 tablet_bootstrap.cc:654] T bc1db73e0d8e44f880057431984bc9f3 P 5f2955b684a2440f94a3d0223eb40dca: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:00.833377 16827 log.cc:824] T bc1db73e0d8e44f880057431984bc9f3 P d24b3090424a46cea6d684dd7f564ab8: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:00.833999 16828 log.cc:824] T bc1db73e0d8e44f880057431984bc9f3 P 5f2955b684a2440f94a3d0223eb40dca: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:00.834944 16827 tablet_bootstrap.cc:492] T bc1db73e0d8e44f880057431984bc9f3 P d24b3090424a46cea6d684dd7f564ab8: No bootstrap required, opened a new log
I20260927 16:42:00.835083 16827 ts_tablet_manager.cc:1397] T bc1db73e0d8e44f880057431984bc9f3 P d24b3090424a46cea6d684dd7f564ab8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:42:00.835616 16828 tablet_bootstrap.cc:492] T bc1db73e0d8e44f880057431984bc9f3 P 5f2955b684a2440f94a3d0223eb40dca: No bootstrap required, opened a new log
I20260927 16:42:00.835803 16828 ts_tablet_manager.cc:1397] T bc1db73e0d8e44f880057431984bc9f3 P 5f2955b684a2440f94a3d0223eb40dca: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:42:00.837951 16827 raft_consensus.cc:348] T bc1db73e0d8e44f880057431984bc9f3 P d24b3090424a46cea6d684dd7f564ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } }
I20260927 16:42:00.838201 16827 raft_consensus.cc:374] T bc1db73e0d8e44f880057431984bc9f3 P d24b3090424a46cea6d684dd7f564ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:00.838272 16827 raft_consensus.cc:729] T bc1db73e0d8e44f880057431984bc9f3 P d24b3090424a46cea6d684dd7f564ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d24b3090424a46cea6d684dd7f564ab8, State: Initialized, Role: FOLLOWER
I20260927 16:42:00.838636 16827 consensus_queue.cc:260] T bc1db73e0d8e44f880057431984bc9f3 P d24b3090424a46cea6d684dd7f564ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } }
I20260927 16:42:00.838968 16828 raft_consensus.cc:348] T bc1db73e0d8e44f880057431984bc9f3 P 5f2955b684a2440f94a3d0223eb40dca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } }
I20260927 16:42:00.839227 16828 raft_consensus.cc:374] T bc1db73e0d8e44f880057431984bc9f3 P 5f2955b684a2440f94a3d0223eb40dca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:00.839275 16814 heartbeater.cc:503] Master 127.0.98.62:34029 was elected leader, sending a full tablet report...
I20260927 16:42:00.839293 16828 raft_consensus.cc:729] T bc1db73e0d8e44f880057431984bc9f3 P 5f2955b684a2440f94a3d0223eb40dca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f2955b684a2440f94a3d0223eb40dca, State: Initialized, Role: FOLLOWER
I20260927 16:42:00.839526 16827 ts_tablet_manager.cc:1428] T bc1db73e0d8e44f880057431984bc9f3 P d24b3090424a46cea6d684dd7f564ab8: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 16:42:00.839717 16828 consensus_queue.cc:260] T bc1db73e0d8e44f880057431984bc9f3 P 5f2955b684a2440f94a3d0223eb40dca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } }
I20260927 16:42:00.840310 16682 heartbeater.cc:503] Master 127.0.98.62:34029 was elected leader, sending a full tablet report...
I20260927 16:42:00.840597 16828 ts_tablet_manager.cc:1428] T bc1db73e0d8e44f880057431984bc9f3 P 5f2955b684a2440f94a3d0223eb40dca: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
W20260927 16:42:00.849498 16683 tablet.cc:2369] T bc1db73e0d8e44f880057431984bc9f3 P 5f2955b684a2440f94a3d0223eb40dca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:42:00.868750 16550 raft_consensus.cc:482] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:42:00.868917 16550 raft_consensus.cc:504] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } }
I20260927 16:42:00.869912 16550 leader_election.cc:288] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f2955b684a2440f94a3d0223eb40dca (127.0.98.2:34077), d24b3090424a46cea6d684dd7f564ab8 (127.0.98.3:41487)
I20260927 16:42:00.873610 16825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:00.870170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36655 (local address 127.0.98.3:41487)
0927 16:42:00.870287 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:00.870291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:00.870358 (+ 67us) server_negotiation.cc:407] Connection header received
0927 16:42:00.870545 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:00.870548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:00.870603 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:00.870675 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:00.870957 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.871631 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.872664 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.873054 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.873125 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:00.873192 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:00.873413 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 16:42:00.873480 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 16:42:00.873612 16547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:00.870098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41487 (local address 127.0.98.1:36655)
0927 16:42:00.870287 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:00.870303 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:00.870405 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:00.870694 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:00.870698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:00.870713 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:00.870868 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:00.870876 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.871733 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.871737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:00.872556 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:00.872564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.873161 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.873166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:00.873283 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:00.873326 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 16:42:00.873381 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 16:42:00.873462 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":4}
I20260927 16:42:00.873649 16823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:00.870163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53251 (local address 127.0.98.2:34077)
0927 16:42:00.870287 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:00.870291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:00.870520 (+ 229us) server_negotiation.cc:407] Connection header received
0927 16:42:00.870643 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:00.870659 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:00.870710 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:00.870791 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:00.871122 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.871793 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.872742 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.873107 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.873203 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:00.873259 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:00.873468 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 16:42:00.873537 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 16:42:00.874101 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:00.870098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34077 (local address 127.0.98.1:53251)
0927 16:42:00.870414 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:00.870425 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:00.870552 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:00.870813 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:00.870815 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:00.870828 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:00.871004 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:00.871011 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.871892 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.871894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:00.872634 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:00.872644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.873240 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.873243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:00.873340 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:00.873356 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 16:42:00.873438 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 16:42:00.873498 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":94,"thread_start_us":89,"threads_started":1}
I20260927 16:42:00.874212 16768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1db73e0d8e44f880057431984bc9f3" candidate_uuid: "de559e6b0a2c43c3a258e764ca9b884e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d24b3090424a46cea6d684dd7f564ab8" is_pre_election: true
I20260927 16:42:00.874502 16768 raft_consensus.cc:2454] T bc1db73e0d8e44f880057431984bc9f3 P d24b3090424a46cea6d684dd7f564ab8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de559e6b0a2c43c3a258e764ca9b884e in term 0.
I20260927 16:42:00.874665 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1db73e0d8e44f880057431984bc9f3" candidate_uuid: "de559e6b0a2c43c3a258e764ca9b884e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f2955b684a2440f94a3d0223eb40dca" is_pre_election: true
I20260927 16:42:00.874949 16636 raft_consensus.cc:2454] T bc1db73e0d8e44f880057431984bc9f3 P 5f2955b684a2440f94a3d0223eb40dca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de559e6b0a2c43c3a258e764ca9b884e in term 0.
I20260927 16:42:00.875022 16430 leader_election.cc:302] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d24b3090424a46cea6d684dd7f564ab8, de559e6b0a2c43c3a258e764ca9b884e; no voters:
I20260927 16:42:00.875288 16550 raft_consensus.cc:2781] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:42:00.875379 16550 raft_consensus.cc:482] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:42:00.875433 16550 raft_consensus.cc:3037] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:00.876437 16550 raft_consensus.cc:504] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } }
I20260927 16:42:00.876781 16550 leader_election.cc:288] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [CANDIDATE]: Term 1 election: Requested vote from peers 5f2955b684a2440f94a3d0223eb40dca (127.0.98.2:34077), d24b3090424a46cea6d684dd7f564ab8 (127.0.98.3:41487)
I20260927 16:42:00.877090 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1db73e0d8e44f880057431984bc9f3" candidate_uuid: "de559e6b0a2c43c3a258e764ca9b884e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f2955b684a2440f94a3d0223eb40dca"
I20260927 16:42:00.877087 16768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1db73e0d8e44f880057431984bc9f3" candidate_uuid: "de559e6b0a2c43c3a258e764ca9b884e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d24b3090424a46cea6d684dd7f564ab8"
I20260927 16:42:00.877192 16636 raft_consensus.cc:3037] T bc1db73e0d8e44f880057431984bc9f3 P 5f2955b684a2440f94a3d0223eb40dca [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:00.877198 16768 raft_consensus.cc:3037] T bc1db73e0d8e44f880057431984bc9f3 P d24b3090424a46cea6d684dd7f564ab8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:00.878328 16636 raft_consensus.cc:2454] T bc1db73e0d8e44f880057431984bc9f3 P 5f2955b684a2440f94a3d0223eb40dca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de559e6b0a2c43c3a258e764ca9b884e in term 1.
I20260927 16:42:00.878328 16768 raft_consensus.cc:2454] T bc1db73e0d8e44f880057431984bc9f3 P d24b3090424a46cea6d684dd7f564ab8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de559e6b0a2c43c3a258e764ca9b884e in term 1.
I20260927 16:42:00.878579 16430 leader_election.cc:302] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d24b3090424a46cea6d684dd7f564ab8, de559e6b0a2c43c3a258e764ca9b884e; no voters:
I20260927 16:42:00.878710 16550 raft_consensus.cc:2781] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:42:00.878895 16550 raft_consensus.cc:686] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [term 1 LEADER]: Becoming Leader. State: Replica: de559e6b0a2c43c3a258e764ca9b884e, State: Running, Role: LEADER
I20260927 16:42:00.879124 16550 consensus_queue.cc:237] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } }
I20260927 16:42:00.881814 16355 catalog_manager.cc:5495] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e reported cstate change: term changed from 0 to 1, leader changed from <none> to de559e6b0a2c43c3a258e764ca9b884e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "de559e6b0a2c43c3a258e764ca9b884e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } health_report { overall_health: HEALTHY } } }
I20260927 16:42:00.903173 16552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:00.899860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.0.98.1:41713)
0927 16:42:00.899988 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:00.899992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:00.900003 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:42:00.900158 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:00.900161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:00.900207 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:00.900303 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:42:00.900546 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.901040 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.901822 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:00.902054 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:00.902146 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:00.902425 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:42:00.902821 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:42:00.902935 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:00.902994 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 16:42:00.903042 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 16:42:00.906901 16636 raft_consensus.cc:1261] T bc1db73e0d8e44f880057431984bc9f3 P 5f2955b684a2440f94a3d0223eb40dca [term 1 FOLLOWER]: Refusing update from remote peer de559e6b0a2c43c3a258e764ca9b884e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:42:00.906901 16768 raft_consensus.cc:1261] T bc1db73e0d8e44f880057431984bc9f3 P d24b3090424a46cea6d684dd7f564ab8 [term 1 FOLLOWER]: Refusing update from remote peer de559e6b0a2c43c3a258e764ca9b884e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:42:00.907683 16550 consensus_queue.cc:1035] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:42:00.907899 16834 consensus_queue.cc:1035] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:42:00.922698 16356 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:42:00.922864 16356 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:42:00.925233 16356 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735: Sending DeleteTablet for 3 replicas of tablet bc1db73e0d8e44f880057431984bc9f3
I20260927 16:42:00.925805 16616 tablet_service.cc:1514] Processing DeleteTablet for tablet bc1db73e0d8e44f880057431984bc9f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:42:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51988
I20260927 16:42:00.925886 16478 tablet_service.cc:1514] Processing DeleteTablet for tablet bc1db73e0d8e44f880057431984bc9f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:42:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46088
I20260927 16:42:00.925971 16748 tablet_service.cc:1514] Processing DeleteTablet for tablet bc1db73e0d8e44f880057431984bc9f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:42:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46626
I20260927 16:42:00.926039 16553 tablet_replica.cc:330] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e: stopping tablet replica
I20260927 16:42:00.926131 16553 raft_consensus.cc:2229] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:42:00.926409 16848 tablet_replica.cc:330] T bc1db73e0d8e44f880057431984bc9f3 P d24b3090424a46cea6d684dd7f564ab8: stopping tablet replica
I20260927 16:42:00.926409 16553 raft_consensus.cc:2258] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:42:00.926693 16848 raft_consensus.cc:2229] T bc1db73e0d8e44f880057431984bc9f3 P d24b3090424a46cea6d684dd7f564ab8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:42:00.926755 16847 tablet_replica.cc:330] T bc1db73e0d8e44f880057431984bc9f3 P 5f2955b684a2440f94a3d0223eb40dca: stopping tablet replica
I20260927 16:42:00.926983 16848 raft_consensus.cc:2258] T bc1db73e0d8e44f880057431984bc9f3 P d24b3090424a46cea6d684dd7f564ab8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:42:00.927017 16847 raft_consensus.cc:2229] T bc1db73e0d8e44f880057431984bc9f3 P 5f2955b684a2440f94a3d0223eb40dca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:42:00.927336 16847 raft_consensus.cc:2258] T bc1db73e0d8e44f880057431984bc9f3 P 5f2955b684a2440f94a3d0223eb40dca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:42:00.927462 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 16394
I20260927 16:42:00.928860 16848 ts_tablet_manager.cc:1905] T bc1db73e0d8e44f880057431984bc9f3 P d24b3090424a46cea6d684dd7f564ab8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:42:00.930824 16848 ts_tablet_manager.cc:1918] T bc1db73e0d8e44f880057431984bc9f3 P d24b3090424a46cea6d684dd7f564ab8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:42:00.930893 16848 log.cc:1196] T bc1db73e0d8e44f880057431984bc9f3 P d24b3090424a46cea6d684dd7f564ab8: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/wal/wals/bc1db73e0d8e44f880057431984bc9f3
I20260927 16:42:00.931167 16848 ts_tablet_manager.cc:1939] T bc1db73e0d8e44f880057431984bc9f3 P d24b3090424a46cea6d684dd7f564ab8: Deleting consensus metadata
I20260927 16:42:00.932075 16847 ts_tablet_manager.cc:1905] T bc1db73e0d8e44f880057431984bc9f3 P 5f2955b684a2440f94a3d0223eb40dca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:42:00.932286 16340 catalog_manager.cc:4841] TS d24b3090424a46cea6d684dd7f564ab8 (127.0.98.3:41487): tablet bc1db73e0d8e44f880057431984bc9f3 (table test-table [id=548013ea07fa46ca9dda461daf797aea]) successfully deleted
W20260927 16:42:00.933561 16340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41713 (error 108)
I20260927 16:42:00.933707 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 16548
I20260927 16:42:00.934993 16551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:00.934760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41713 (local address 127.0.0.1:46098)
0927 16:42:00.934872 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:00.934933 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 16:42:00.935124 16340 catalog_manager.cc:4584] TS de559e6b0a2c43c3a258e764ca9b884e (127.0.98.1:41713): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bc1db73e0d8e44f880057431984bc9f3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41713: connect: Connection refused (error 111)
I20260927 16:42:00.939391 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 16685
I20260927 16:42:00.940382 16551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:00.939617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34077 (local address 127.0.0.1:52000)
0927 16:42:00.940278 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:00.940325 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":597}
I20260927 16:42:00.944154 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 16323
I20260927 16:42:20.950614 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34029
--webserver_interface=127.0.98.62
--webserver_port=32829
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:42:21.051534 16850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:21.051785 16850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:21.051849 16850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:21.056046 16850 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 16:42:21.056087 16850 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 16:42:21.056102 16850 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 16:42:21.056115 16850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:42:21.056165 16850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:21.056198 16850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:21.056231 16850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:42:21.056278 16850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:42:21.061002 16850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=32829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:21.061851 16850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:21.062726 16850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:21.070701 16858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:21.070725 16855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:21.070772 16856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:21.070829 16850 server_base.cc:1034] running on GCE node
I20260927 16:42:21.071552 16850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:21.072501 16850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:21.073654 16850 hybrid_clock.cc:648] HybridClock initialized: now 1790527341073630 us; error 37 us; skew 500 ppm
Sep 27 16:42:21 dist-test-slave-z40b krb5kdc[15776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:21.076334 16850 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:42:21.077152 16850 webserver.cc:460] Webserver started at http://127.0.98.62:32829/ using document root <none> and password file <none>
I20260927 16:42:21.077665 16850 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:21.077751 16850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:21.081125 16850 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:42:21.082984 16865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:21.084153 16850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:42:21.084293 16850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "3efdfc7c9e424048a60b502265e87735"
format_stamp: "Formatted at 2026-09-27 16:41:39 on dist-test-slave-z40b"
server_key: "0ae97af13245a9eb3269e28140fdbbf1"
server_key_iv: "304733397a6a58c78a2b838dba9acf39"
server_key_version: "encryptionkey@0"
I20260927 16:42:21.084805 16850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:21.098284 16850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:21.101295 16850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:21.101476 16850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:21.108069 16850 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34029
I20260927 16:42:21.108348 16917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34029 every 8 connection(s)
I20260927 16:42:21.109072 16850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:42:21.113229 16918 sys_catalog.cc:263] Verifying existing consensus state
I20260927 16:42:21.114770 16918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735: Bootstrap starting.
I20260927 16:42:21.116559 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 16850
I20260927 16:42:21.117060 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41713
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34029
--webserver_port=43089
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:42:21.132318 16918 log.cc:824] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:21.142979 16918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:42:21.143388 16918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735: Bootstrap complete.
I20260927 16:42:21.147270 16918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efdfc7c9e424048a60b502265e87735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } }
I20260927 16:42:21.147634 16918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdfc7c9e424048a60b502265e87735, State: Initialized, Role: FOLLOWER
I20260927 16:42:21.148101 16918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efdfc7c9e424048a60b502265e87735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } }
I20260927 16:42:21.148231 16918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:42:21.148317 16918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:42:21.148425 16918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [term 2 FOLLOWER]: Advancing to term 3
I20260927 16:42:21.150187 16918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efdfc7c9e424048a60b502265e87735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } }
I20260927 16:42:21.150522 16918 leader_election.cc:302] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3efdfc7c9e424048a60b502265e87735; no voters:
I20260927 16:42:21.150796 16918 leader_election.cc:288] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 16:42:21.150916 16922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 16:42:21.151140 16922 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [term 3 LEADER]: Becoming Leader. State: Replica: 3efdfc7c9e424048a60b502265e87735, State: Running, Role: LEADER
I20260927 16:42:21.151470 16922 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efdfc7c9e424048a60b502265e87735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } }
I20260927 16:42:21.151958 16918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:42:21.152750 16923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3efdfc7c9e424048a60b502265e87735" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efdfc7c9e424048a60b502265e87735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } } }
I20260927 16:42:21.152871 16923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:21.153172 16924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3efdfc7c9e424048a60b502265e87735. Latest consensus state: current_term: 3 leader_uuid: "3efdfc7c9e424048a60b502265e87735" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3efdfc7c9e424048a60b502265e87735" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } } }
I20260927 16:42:21.153245 16924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735 [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:21.153570 16931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:42:21.158308 16931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:42:21.158934 16931 catalog_manager.cc:1259] Loaded cluster ID: 843cf9e83d594d3fbdce518e6f4e9f17
I20260927 16:42:21.159008 16931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:42:21.161660 16931 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:42:21.162370 16931 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735: Loaded TSK: 0
I20260927 16:42:21.162447 16931 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3efdfc7c9e424048a60b502265e87735: Loaded TSK: 1
F20260927 16:42:21.163130 16931 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f08e20969ad google::LogMessage::Fail() at ??:0
@ 0x7f08e209988a google::LogMessage::SendToLog() at ??:0
@ 0x7f08e20964de google::LogMessage::Flush() at ??:0
@ 0x7f08e2097a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f08e4d5a18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f08e4d52f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f08ec9fb7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f08ec9d599c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f08ec9d6178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f08eca17df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f08eca3682b std::function<>::operator()() at ??:0
@ 0x7f08ec9d5b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f08ec9d6f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f08ec9d3579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f08eca17640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f08e42e83d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f08e42e8c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f08e42ea3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f08e42d9512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f08e591d6db start_thread at ??:0
@ 0x7f08e0b9461f clone at ??:0
W20260927 16:42:21.249303 16920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:21.249508 16920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:21.249548 16920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:21.253762 16920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:21.253799 16920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:21.253854 16920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:42:21.259119 16920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41713
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43089
--tserver_master_addrs=127.0.98.62:34029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:21.260277 16920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:21.261250 16920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:21.270612 16965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:21.270617 16968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:21.270869 16966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:21.270916 16920 server_base.cc:1034] running on GCE node
I20260927 16:42:21.271366 16920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:21.271960 16920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:21.273139 16920 hybrid_clock.cc:648] HybridClock initialized: now 1790527341273115 us; error 42 us; skew 500 ppm
Sep 27 16:42:21 dist-test-slave-z40b krb5kdc[15776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:21.275858 16920 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:42:21.276665 16920 webserver.cc:460] Webserver started at http://127.0.98.1:43089/ using document root <none> and password file <none>
I20260927 16:42:21.277196 16920 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:21.277256 16920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:21.281031 16920 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:42:21.283178 16975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:21.284415 16920 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:42:21.284541 16920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "de559e6b0a2c43c3a258e764ca9b884e"
format_stamp: "Formatted at 2026-09-27 16:41:39 on dist-test-slave-z40b"
server_key: "03491d74f493144878b5f3a197841d2c"
server_key_iv: "7d133fc6fe680e75dc3838c469ee0ab2"
server_key_version: "encryptionkey@0"
I20260927 16:42:21.285073 16920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:21.300920 16920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:21.303900 16920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:21.304081 16920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:21.304646 16920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:42:21.305706 16982 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 16:42:21.311436 16920 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 16:42:21.311523 16920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 16:42:21.311605 16920 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 16:42:21.313285 16920 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 16:42:21.313356 16920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 16:42:21.313627 16982 tablet_bootstrap.cc:492] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e: Bootstrap starting.
I20260927 16:42:21.319934 16982 log.cc:824] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:21.323364 16920 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41713
I20260927 16:42:21.324216 16920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:42:21.325284 17090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41713 every 8 connection(s)
I20260927 16:42:21.329450 16982 tablet_bootstrap.cc:492] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:42:21.329888 16982 tablet_bootstrap.cc:492] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e: Bootstrap complete.
I20260927 16:42:21.330291 16982 ts_tablet_manager.cc:1397] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.001s
I20260927 16:42:21.333393 16982 raft_consensus.cc:348] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } }
I20260927 16:42:21.333865 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 16920
I20260927 16:42:21.333931 16982 raft_consensus.cc:729] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: de559e6b0a2c43c3a258e764ca9b884e, State: Initialized, Role: FOLLOWER
I20260927 16:42:21.334393 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34077
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34029
--webserver_port=34781
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:42:21.334483 16982 consensus_queue.cc:260] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } }
I20260927 16:42:21.335528 16982 ts_tablet_manager.cc:1428] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
W20260927 16:42:21.413980 17093 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:42:21.326187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.1:43057)
0927 16:42:21.326605 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:21.326654 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:21.327588 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:21.413248 (+ 85660us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":119,"threads_started":1}
W20260927 16:42:21.414572 17091 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34029 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34029: Client connection negotiation failed: client connection to 127.0.98.62:34029: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 16:42:21.415340 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:21.415041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.1:35627)
0927 16:42:21.415182 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:21.415273 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 16:42:21.416060 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:21.415850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.1:50117)
0927 16:42:21.415957 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:21.416000 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 16:42:21.416253 17091 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 16:42:21.461388 17095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:21.461633 17095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:21.461678 17095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:21.465624 17095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:21.465674 17095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:21.465765 17095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:42:21.470877 17095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34781
--tserver_master_addrs=127.0.98.62:34029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:21.471870 17095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:21.472862 17095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:21.482065 17101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:21.482079 17103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:21.482126 17100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:21.482287 17095 server_base.cc:1034] running on GCE node
I20260927 16:42:21.482721 17095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:21.483338 17095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:21.484498 17095 hybrid_clock.cc:648] HybridClock initialized: now 1790527341484484 us; error 25 us; skew 500 ppm
Sep 27 16:42:21 dist-test-slave-z40b krb5kdc[15776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:21.487089 17095 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:42:21.487964 17095 webserver.cc:460] Webserver started at http://127.0.98.2:34781/ using document root <none> and password file <none>
I20260927 16:42:21.488528 17095 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:21.488610 17095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:21.493206 17095 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 16:42:21.495299 17110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:21.496464 17095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:42:21.496605 17095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "5f2955b684a2440f94a3d0223eb40dca"
format_stamp: "Formatted at 2026-09-27 16:41:39 on dist-test-slave-z40b"
server_key: "349622560a4f0dd371097cac7a4a7f74"
server_key_iv: "c29871915d99e1628ba4d7409b8d6f11"
server_key_version: "encryptionkey@0"
I20260927 16:42:21.497144 17095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:21.514011 17095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:21.516664 17095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:21.516852 17095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:21.517396 17095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:42:21.518396 17117 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260927 16:42:21.522097 17117 ts_tablet_manager.cc:1852] T bc1db73e0d8e44f880057431984bc9f3 P 5f2955b684a2440f94a3d0223eb40dca: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 16:42:21.522498 17117 ts_tablet_manager.cc:1905] T bc1db73e0d8e44f880057431984bc9f3 P 5f2955b684a2440f94a3d0223eb40dca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:42:21.526126 17117 ts_tablet_manager.cc:1918] T bc1db73e0d8e44f880057431984bc9f3 P 5f2955b684a2440f94a3d0223eb40dca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:42:21.526263 17117 log.cc:1196] T bc1db73e0d8e44f880057431984bc9f3 P 5f2955b684a2440f94a3d0223eb40dca: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/wal/wals/bc1db73e0d8e44f880057431984bc9f3
I20260927 16:42:21.526587 17117 ts_tablet_manager.cc:1939] T bc1db73e0d8e44f880057431984bc9f3 P 5f2955b684a2440f94a3d0223eb40dca: Deleting consensus metadata
I20260927 16:42:21.526911 17095 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 16:42:21.526979 17095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260927 16:42:21.527050 17095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:42:21.527086 17095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:21.535775 17095 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34077
I20260927 16:42:21.535832 17224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34077 every 8 connection(s)
I20260927 16:42:21.536597 17095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:42:21.539889 17227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:21.538535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.2:41159)
0927 16:42:21.539044 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:21.539146 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":132,"threads_started":1}
W20260927 16:42:21.540279 17225 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34029 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34029: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
I20260927 16:42:21.540779 17227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:21.540589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.2:41497)
0927 16:42:21.540687 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:21.540730 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 16:42:21.541373 17227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:21.541196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.2:46481)
0927 16:42:21.541279 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:21.541317 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 16:42:21.541570 17225 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 16:42:21.541790 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 17095
I20260927 16:42:21.542192 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41487
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34029
--webserver_port=40561
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:42:21.646531 17228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:21.646751 17228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:21.646790 17228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:21.650738 17228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:21.650787 17228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:21.650839 17228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:42:21.655740 17228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40561
--tserver_master_addrs=127.0.98.62:34029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:21.656746 17228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:21.657708 17228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:21.666538 17233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:21.666539 17236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:21.666517 17234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:21.666618 17228 server_base.cc:1034] running on GCE node
I20260927 16:42:21.667248 17228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:21.667847 17228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:21.669024 17228 hybrid_clock.cc:648] HybridClock initialized: now 1790527341668999 us; error 46 us; skew 500 ppm
Sep 27 16:42:21 dist-test-slave-z40b krb5kdc[15776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:21.671700 17228 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:42:21.672472 17228 webserver.cc:460] Webserver started at http://127.0.98.3:40561/ using document root <none> and password file <none>
I20260927 16:42:21.673009 17228 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:21.673092 17228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:21.676910 17228 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:42:21.679265 17243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:21.680480 17228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:42:21.680611 17228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "d24b3090424a46cea6d684dd7f564ab8"
format_stamp: "Formatted at 2026-09-27 16:41:39 on dist-test-slave-z40b"
server_key: "41a6723c1d6104735193e1463a766841"
server_key_iv: "c4e523161339659862b8fb02485f173a"
server_key_version: "encryptionkey@0"
I20260927 16:42:21.681110 17228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:21.699059 17228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:21.701881 17228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:21.702067 17228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:21.702625 17228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:42:21.703470 17228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:42:21.703517 17228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:21.703579 17228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:42:21.703611 17228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:21.712110 17228 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41487
I20260927 16:42:21.712152 17356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41487 every 8 connection(s)
I20260927 16:42:21.713023 17228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:42:21.716451 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:21.714974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.3:44657)
0927 16:42:21.715689 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:21.715791 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":106,"threads_started":1}
W20260927 16:42:21.716876 17357 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34029 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34029: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
I20260927 16:42:21.717365 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:21.717187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.3:52725)
0927 16:42:21.717278 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:21.717319 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 16:42:21.717857 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 17228
I20260927 16:42:21.718067 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:21.717843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.3:40297)
0927 16:42:21.717957 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:21.718027 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 16:42:21.718233 17357 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 16:42:22.417174 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:22.416779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.1:33695)
0927 16:42:22.417028 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:22.417076 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":68,"threads_started":1}
I20260927 16:42:22.542342 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:22.542018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.2:35607)
0927 16:42:22.542223 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:22.542267 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":66,"threads_started":1}
I20260927 16:42:22.719137 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:22.718784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.3:51915)
0927 16:42:22.718996 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:22.719047 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":61,"threads_started":1}
I20260927 16:42:23.181037 17364 raft_consensus.cc:482] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:42:23.181247 17364 raft_consensus.cc:504] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } }
I20260927 16:42:23.182384 17364 leader_election.cc:288] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f2955b684a2440f94a3d0223eb40dca (127.0.98.2:34077), d24b3090424a46cea6d684dd7f564ab8 (127.0.98.3:41487)
Sep 27 16:42:23 dist-test-slave-z40b krb5kdc[15776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 27 16:42:23 dist-test-slave-z40b krb5kdc[15776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 16:42:23.197834 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:23.182494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34077 (local address 127.0.98.1:38701)
0927 16:42:23.182767 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:23.182780 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:23.182873 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:23.184446 (+ 1573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:23.184451 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:23.185083 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:42:23.185828 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:23.185849 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:23.187688 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:23.187692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:23.189395 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:23.189406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:23.190345 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:23.190348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:23.190631 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:23.191397 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:42:23.191417 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:42:23.193667 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:42:23.195669 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:23.195672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:23.195673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:23.195951 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:23.196297 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:23.196300 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:23.196301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:23.196449 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:23.196999 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:42:23.197004 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:42:23.197195 (+ 191us) client_negotiation.cc:769] Sending connection context
0927 16:42:23.197362 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 16:42:23.197653 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":441,"spinlock_wait_cycles":11648,"thread_start_us":61,"threads_started":1}
I20260927 16:42:23.197830 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:23.182803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52749 (local address 127.0.98.3:41487)
0927 16:42:23.183122 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:23.183145 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:23.183186 (+ 41us) server_negotiation.cc:407] Connection header received
0927 16:42:23.183796 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:23.183802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:23.184108 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:23.184441 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:23.186004 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:23.187528 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:23.189515 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:23.190224 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:23.190383 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:23.193479 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:23.193484 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:23.193497 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:23.193527 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:23.195463 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:23.196060 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:23.196066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:23.196084 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:23.196199 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:23.196579 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:23.196583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:23.196585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:23.196893 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:23.197015 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:23.197458 (+ 443us) server_negotiation.cc:299] Negotiation successful
0927 16:42:23.197674 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":83,"threads_started":1}
I20260927 16:42:23.197877 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:23.182610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41487 (local address 127.0.98.1:52749)
0927 16:42:23.183035 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:23.183046 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:23.183125 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:23.184445 (+ 1320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:23.184451 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:23.185083 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:42:23.185828 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:23.185849 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:23.187688 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:23.187692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:23.189395 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:23.189406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:23.190345 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:23.190348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:23.190631 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:23.191410 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:42:23.191422 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:42:23.193255 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:42:23.195573 (+ 2318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:23.195578 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:23.195592 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:23.195950 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:23.196325 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:23.196327 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:23.196329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:23.196458 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:23.197023 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:42:23.197026 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:42:23.197194 (+ 168us) client_negotiation.cc:769] Sending connection context
0927 16:42:23.197361 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 16:42:23.197725 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":503,"spinlock_wait_cycles":44544,"thread_start_us":61,"threads_started":1}
I20260927 16:42:23.197881 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:23.182678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38701 (local address 127.0.98.2:34077)
0927 16:42:23.182957 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:23.182972 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:23.183014 (+ 42us) server_negotiation.cc:407] Connection header received
0927 16:42:23.183717 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:23.183723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:23.184123 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:23.184404 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:23.186004 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:23.187528 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:23.189515 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:23.190224 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:23.190383 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:23.193823 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:23.193827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:23.193836 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:23.193859 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:23.195561 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:23.196061 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:23.196066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:23.196071 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:23.196193 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:23.196537 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:23.196540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:23.196551 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:23.196879 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:23.196989 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:23.197459 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 16:42:23.197710 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":78,"threads_started":1}
W20260927 16:42:23.200230 16978 leader_election.cc:341] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5f2955b684a2440f94a3d0223eb40dca (127.0.98.2:34077): Not found: Tablet not found: bc1db73e0d8e44f880057431984bc9f3
W20260927 16:42:23.200522 16976 leader_election.cc:341] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d24b3090424a46cea6d684dd7f564ab8 (127.0.98.3:41487): Not found: Tablet not found: bc1db73e0d8e44f880057431984bc9f3
I20260927 16:42:23.200613 16976 leader_election.cc:302] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de559e6b0a2c43c3a258e764ca9b884e; no voters: 5f2955b684a2440f94a3d0223eb40dca, d24b3090424a46cea6d684dd7f564ab8
I20260927 16:42:23.200809 17364 raft_consensus.cc:2726] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:42:23.418277 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:23.417912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.1:39699)
0927 16:42:23.418091 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:23.418163 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 16:42:23.543442 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:23.542974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.2:38793)
0927 16:42:23.543281 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:23.543345 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":91,"threads_started":1}
I20260927 16:42:23.720172 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:23.719815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.3:37093)
0927 16:42:23.720032 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:23.720087 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":69,"threads_started":1}
I20260927 16:42:24.419523 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:24.419109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.1:42523)
0927 16:42:24.419375 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:24.419430 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":112,"threads_started":1}
I20260927 16:42:24.544570 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:24.544135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.2:48171)
0927 16:42:24.544400 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:24.544451 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":98,"threads_started":1}
I20260927 16:42:24.721308 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:24.720889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.3:59829)
0927 16:42:24.721143 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:24.721193 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":87,"threads_started":1}
I20260927 16:42:24.850325 17374 raft_consensus.cc:482] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:42:24.850461 17374 raft_consensus.cc:504] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } }
I20260927 16:42:24.850914 17374 leader_election.cc:288] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f2955b684a2440f94a3d0223eb40dca (127.0.98.2:34077), d24b3090424a46cea6d684dd7f564ab8 (127.0.98.3:41487)
W20260927 16:42:24.851711 16976 leader_election.cc:341] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d24b3090424a46cea6d684dd7f564ab8 (127.0.98.3:41487): Not found: Tablet not found: bc1db73e0d8e44f880057431984bc9f3
W20260927 16:42:24.851801 16978 leader_election.cc:341] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5f2955b684a2440f94a3d0223eb40dca (127.0.98.2:34077): Not found: Tablet not found: bc1db73e0d8e44f880057431984bc9f3
I20260927 16:42:24.851902 16978 leader_election.cc:302] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de559e6b0a2c43c3a258e764ca9b884e; no voters: 5f2955b684a2440f94a3d0223eb40dca, d24b3090424a46cea6d684dd7f564ab8
I20260927 16:42:24.852065 17374 raft_consensus.cc:2726] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:42:25.420553 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:25.420177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.1:52947)
0927 16:42:25.420399 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:25.420449 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":67,"threads_started":1}
I20260927 16:42:25.545573 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:25.545202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.2:60681)
0927 16:42:25.545414 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:25.545464 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":65,"threads_started":1}
I20260927 16:42:25.722350 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:25.721992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.3:39121)
0927 16:42:25.722216 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:25.722260 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":86,"threads_started":1}
I20260927 16:42:26.421872 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:26.421468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.1:32979)
0927 16:42:26.421732 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:26.421783 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":80,"threads_started":1}
I20260927 16:42:26.546604 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:26.546240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.2:45239)
0927 16:42:26.546463 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:26.546514 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":70,"threads_started":1}
I20260927 16:42:26.723876 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:26.723293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.3:57323)
0927 16:42:26.723714 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:26.723786 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":66,"threads_started":1}
I20260927 16:42:27.423094 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:27.422689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.1:38051)
0927 16:42:27.422940 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:27.422990 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":88,"threads_started":1}
I20260927 16:42:27.547643 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:27.547300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.2:48945)
0927 16:42:27.547509 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:27.547562 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":66,"threads_started":1}
I20260927 16:42:27.589973 17383 raft_consensus.cc:482] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:42:27.590088 17383 raft_consensus.cc:504] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } }
I20260927 16:42:27.590435 17383 leader_election.cc:288] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f2955b684a2440f94a3d0223eb40dca (127.0.98.2:34077), d24b3090424a46cea6d684dd7f564ab8 (127.0.98.3:41487)
W20260927 16:42:27.591146 16978 leader_election.cc:341] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5f2955b684a2440f94a3d0223eb40dca (127.0.98.2:34077): Not found: Tablet not found: bc1db73e0d8e44f880057431984bc9f3
W20260927 16:42:27.591325 16976 leader_election.cc:341] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d24b3090424a46cea6d684dd7f564ab8 (127.0.98.3:41487): Not found: Tablet not found: bc1db73e0d8e44f880057431984bc9f3
I20260927 16:42:27.591421 16976 leader_election.cc:302] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de559e6b0a2c43c3a258e764ca9b884e; no voters: 5f2955b684a2440f94a3d0223eb40dca, d24b3090424a46cea6d684dd7f564ab8
I20260927 16:42:27.591557 17383 raft_consensus.cc:2726] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:42:27.725072 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:27.724650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.3:35805)
0927 16:42:27.724908 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:27.724962 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":94,"threads_started":1}
I20260927 16:42:28.424206 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:28.423832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.1:41903)
0927 16:42:28.424064 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:28.424113 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":74,"threads_started":1}
I20260927 16:42:28.548636 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:28.548261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.2:38753)
0927 16:42:28.548511 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:28.548560 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":61,"threads_started":1}
I20260927 16:42:28.726284 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:28.725854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.3:42665)
0927 16:42:28.726140 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:28.726193 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":67,"threads_started":1}
I20260927 16:42:29.425228 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:29.424863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.1:41933)
0927 16:42:29.425091 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:29.425139 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":77,"threads_started":1}
I20260927 16:42:29.549696 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:29.549247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.2:51089)
0927 16:42:29.549569 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:29.549618 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":71,"threads_started":1}
I20260927 16:42:29.727515 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:29.727058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.3:37159)
0927 16:42:29.727370 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:29.727423 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":71,"threads_started":1}
I20260927 16:42:30.426375 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:30.425955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.1:44207)
0927 16:42:30.426197 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:30.426244 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":92,"threads_started":1}
I20260927 16:42:30.550856 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:30.550458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.2:37335)
0927 16:42:30.550719 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:30.550769 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":101,"threads_started":1}
I20260927 16:42:30.728628 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:30.728215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.3:36753)
0927 16:42:30.728465 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:30.728530 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":84,"threads_started":1}
I20260927 16:42:31.427723 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:31.427344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.1:42877)
0927 16:42:31.427580 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:31.427632 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":69,"threads_started":1}
I20260927 16:42:31.551898 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:31.551529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.2:57949)
0927 16:42:31.551747 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:31.551797 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":64,"threads_started":1}
I20260927 16:42:31.729764 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:31.729322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.3:45021)
0927 16:42:31.729590 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:31.729643 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":83,"threads_started":1}
I20260927 16:42:32.369115 17397 raft_consensus.cc:482] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:42:32.369259 17397 raft_consensus.cc:504] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2955b684a2440f94a3d0223eb40dca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34077 } } peers { permanent_uuid: "d24b3090424a46cea6d684dd7f564ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41487 } } peers { permanent_uuid: "de559e6b0a2c43c3a258e764ca9b884e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41713 } }
I20260927 16:42:32.369686 17397 leader_election.cc:288] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f2955b684a2440f94a3d0223eb40dca (127.0.98.2:34077), d24b3090424a46cea6d684dd7f564ab8 (127.0.98.3:41487)
W20260927 16:42:32.370385 16976 leader_election.cc:341] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d24b3090424a46cea6d684dd7f564ab8 (127.0.98.3:41487): Not found: Tablet not found: bc1db73e0d8e44f880057431984bc9f3
W20260927 16:42:32.370486 16978 leader_election.cc:341] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5f2955b684a2440f94a3d0223eb40dca (127.0.98.2:34077): Not found: Tablet not found: bc1db73e0d8e44f880057431984bc9f3
I20260927 16:42:32.370599 16978 leader_election.cc:302] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de559e6b0a2c43c3a258e764ca9b884e; no voters: 5f2955b684a2440f94a3d0223eb40dca, d24b3090424a46cea6d684dd7f564ab8
I20260927 16:42:32.370738 17397 raft_consensus.cc:2726] T bc1db73e0d8e44f880057431984bc9f3 P de559e6b0a2c43c3a258e764ca9b884e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:42:32.428779 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:32.428411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.1:56533)
0927 16:42:32.428643 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:32.428691 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":78,"threads_started":1}
I20260927 16:42:32.552850 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:32.552509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.2:55691)
0927 16:42:32.552721 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:32.552767 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":62,"threads_started":1}
I20260927 16:42:32.730858 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:32.730498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.3:39543)
0927 16:42:32.730707 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:32.730770 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":65,"threads_started":1}
I20260927 16:42:33.429915 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:33.429419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.1:38401)
0927 16:42:33.429744 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:33.429809 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":77,"threads_started":1}
I20260927 16:42:33.553834 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:33.553467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.2:57189)
0927 16:42:33.553703 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:33.553752 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":64,"threads_started":1}
I20260927 16:42:33.731959 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:33.731582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.3:36311)
0927 16:42:33.731808 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:33.731858 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":67,"threads_started":1}
I20260927 16:42:34.430976 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:34.430610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.1:48861)
0927 16:42:34.430836 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:34.430886 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":66,"threads_started":1}
I20260927 16:42:34.554805 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:34.554446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.2:58829)
0927 16:42:34.554669 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:34.554718 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":67,"threads_started":1}
I20260927 16:42:34.733063 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:34.732655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.3:37965)
0927 16:42:34.732917 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:34.732967 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":82,"threads_started":1}
I20260927 16:42:35.432045 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:35.431667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.1:44515)
0927 16:42:35.431908 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:35.431954 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":81,"threads_started":1}
I20260927 16:42:35.555961 17408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:35.555576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.2:39887)
0927 16:42:35.555823 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:35.555877 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":79,"threads_started":1}
I20260927 16:42:35.734284 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:35.733840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.3:48957)
0927 16:42:35.734144 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:35.734194 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":69,"threads_started":1}
I20260927 16:42:36.433382 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:36.432981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.1:40407)
0927 16:42:36.433243 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:36.433296 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":118,"threads_started":1}
I20260927 16:42:36.557097 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:36.556704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.98.2:51263)
0927 16:42:36.556932 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:36.556997 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":68,"threads_started":1}
I20260927 16:42:36.721089 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 16920
I20260927 16:42:36.726848 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 17095
I20260927 16:42:36.732113 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 17228
2026-09-27T16:42:36Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58938 ms)
[----------] 30 tests from SecurityITest (235210 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:42:36 dist-test-slave-z40b krb5kdc[17414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:42:36 dist-test-slave-z40b krb5kdc[17414](info): set up 2 sockets
Sep 27 16:42:36 dist-test-slave-z40b krb5kdc[17414](info): commencing operation
krb5kdc: starting...
W20260927 16:42:37.751192 392 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:42:37 dist-test-slave-z40b krb5kdc[17414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:42:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:42:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:42:37.866439 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36865
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:42:37.973467 17427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:37.973730 17427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:37.973793 17427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:37.977679 17427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:42:37.977759 17427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:37.977797 17427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:37.977854 17427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:42:37.977895 17427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:42:37.982766 17427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:37.983778 17427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:37.984712 17427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:37.993747 17432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:37.993777 17435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:37.993777 17433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:37.993881 17427 server_base.cc:1034] running on GCE node
I20260927 16:42:37.994612 17427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:37.995663 17427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:37.996847 17427 hybrid_clock.cc:648] HybridClock initialized: now 1790527357996813 us; error 56 us; skew 500 ppm
Sep 27 16:42:37 dist-test-slave-z40b krb5kdc[17414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:37.999500 17427 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:42:38.000317 17427 webserver.cc:460] Webserver started at http://127.0.98.62:36191/ using document root <none> and password file <none>
I20260927 16:42:38.000871 17427 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:38.000959 17427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:38.001185 17427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:38.003165 17427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "f1dd72b9f75142acb77abbac596b5154"
format_stamp: "Formatted at 2026-09-27 16:42:37 on dist-test-slave-z40b"
server_key: "fc975ebb0e422fd495a6bd9b1e89474c"
server_key_iv: "f224e5cb2f88db3d8154d7f67158033f"
server_key_version: "encryptionkey@0"
I20260927 16:42:38.003764 17427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "f1dd72b9f75142acb77abbac596b5154"
format_stamp: "Formatted at 2026-09-27 16:42:37 on dist-test-slave-z40b"
server_key: "fc975ebb0e422fd495a6bd9b1e89474c"
server_key_iv: "f224e5cb2f88db3d8154d7f67158033f"
server_key_version: "encryptionkey@0"
I20260927 16:42:38.007303 17427 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 16:42:38.009567 17442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:38.010671 17427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:42:38.010803 17427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "f1dd72b9f75142acb77abbac596b5154"
format_stamp: "Formatted at 2026-09-27 16:42:37 on dist-test-slave-z40b"
server_key: "fc975ebb0e422fd495a6bd9b1e89474c"
server_key_iv: "f224e5cb2f88db3d8154d7f67158033f"
server_key_version: "encryptionkey@0"
I20260927 16:42:38.010907 17427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:38.034217 17427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:38.037401 17427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:38.037599 17427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:38.044567 17427 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36865
I20260927 16:42:38.044672 17494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36865 every 8 connection(s)
I20260927 16:42:38.045487 17427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:42:38.048116 17495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:38.052273 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 17427
I20260927 16:42:38.052408 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:42:38.052700 392 external_mini_cluster.cc:1428] Setting key d6bd7491246805febf8c97b134a36d66
I20260927 16:42:38.053674 17495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1dd72b9f75142acb77abbac596b5154: Bootstrap starting.
I20260927 16:42:38.056277 17495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1dd72b9f75142acb77abbac596b5154: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:38.057106 17495 log.cc:824] T 00000000000000000000000000000000 P f1dd72b9f75142acb77abbac596b5154: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:38.059028 17495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1dd72b9f75142acb77abbac596b5154: No bootstrap required, opened a new log
Sep 27 16:42:38 dist-test-slave-z40b krb5kdc[17414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:42:38.063067 17495 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1dd72b9f75142acb77abbac596b5154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1dd72b9f75142acb77abbac596b5154" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36865 } }
I20260927 16:42:38.063381 17495 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1dd72b9f75142acb77abbac596b5154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:38.063445 17495 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1dd72b9f75142acb77abbac596b5154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1dd72b9f75142acb77abbac596b5154, State: Initialized, Role: FOLLOWER
I20260927 16:42:38.063977 17495 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1dd72b9f75142acb77abbac596b5154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1dd72b9f75142acb77abbac596b5154" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36865 } }
I20260927 16:42:38.064131 17495 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1dd72b9f75142acb77abbac596b5154 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:42:38.064198 17495 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1dd72b9f75142acb77abbac596b5154 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:42:38.064283 17495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1dd72b9f75142acb77abbac596b5154 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:38.065307 17495 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1dd72b9f75142acb77abbac596b5154 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1dd72b9f75142acb77abbac596b5154" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36865 } }
I20260927 16:42:38.065689 17495 leader_election.cc:302] T 00000000000000000000000000000000 P f1dd72b9f75142acb77abbac596b5154 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1dd72b9f75142acb77abbac596b5154; no voters:
I20260927 16:42:38.066000 17495 leader_election.cc:288] T 00000000000000000000000000000000 P f1dd72b9f75142acb77abbac596b5154 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:42:38.066104 17500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1dd72b9f75142acb77abbac596b5154 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:42:38.066344 17500 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1dd72b9f75142acb77abbac596b5154 [term 1 LEADER]: Becoming Leader. State: Replica: f1dd72b9f75142acb77abbac596b5154, State: Running, Role: LEADER
I20260927 16:42:38.066699 17500 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1dd72b9f75142acb77abbac596b5154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1dd72b9f75142acb77abbac596b5154" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36865 } }
I20260927 16:42:38.067286 17495 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1dd72b9f75142acb77abbac596b5154 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:42:38.068346 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:38.054027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40074 (local address 127.0.98.62:36865)
0927 16:42:38.054418 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:38.054435 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:38.054474 (+ 39us) server_negotiation.cc:407] Connection header received
0927 16:42:38.055326 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:38.055348 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:38.055723 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:38.056092 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:38.056829 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.058165 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.059019 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.059700 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.059859 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:38.062661 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:38.062667 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:38.062683 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:38.062712 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:38.064733 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:38.065124 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:38.065132 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:38.065138 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:38.065287 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:38.065571 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:38.065574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:38.065576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:38.066085 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:38.066850 (+ 765us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:38.067017 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 16:42:38.067326 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":99,"threads_started":1}
I20260927 16:42:38.068466 17501 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1dd72b9f75142acb77abbac596b5154 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1dd72b9f75142acb77abbac596b5154" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1dd72b9f75142acb77abbac596b5154" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36865 } } }
I20260927 16:42:38.068465 17502 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1dd72b9f75142acb77abbac596b5154 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1dd72b9f75142acb77abbac596b5154. Latest consensus state: current_term: 1 leader_uuid: "f1dd72b9f75142acb77abbac596b5154" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1dd72b9f75142acb77abbac596b5154" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36865 } } }
I20260927 16:42:38.068750 17502 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1dd72b9f75142acb77abbac596b5154 [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:38.068750 17501 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1dd72b9f75142acb77abbac596b5154 [sys.catalog]: This master's current role is: LEADER
W20260927 16:42:38.072170 17515 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f1dd72b9f75142acb77abbac596b5154: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:42:38.072247 17515 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:42:38.072321 17510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:42:38.072976 17510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:42:38.078380 17510 catalog_manager.cc:1347] Generated new cluster ID: 8f16d162cd354d7d8ea4fc2f27a00033
I20260927 16:42:38.078438 17510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:42:38.090998 17510 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:42:38.091905 17510 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:42:38.098229 17510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1dd72b9f75142acb77abbac596b5154: Generated new TSK 0
I20260927 16:42:38.098814 17510 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:42:38.130105 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36865
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:42:38.237746 17521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:38.237983 17521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:38.238025 17521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:38.241946 17521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:38.241997 17521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:38.242051 17521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:42:38.247226 17521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:38.248224 17521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:38.249197 17521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:38.258765 17529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:38.258819 17521 server_base.cc:1034] running on GCE node
W20260927 16:42:38.258781 17527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:38.258771 17526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:38.259418 17521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:38.259971 17521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:38.261161 17521 hybrid_clock.cc:648] HybridClock initialized: now 1790527358261147 us; error 25 us; skew 500 ppm
Sep 27 16:42:38 dist-test-slave-z40b krb5kdc[17414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:38.264094 17521 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:42:38.264918 17521 webserver.cc:460] Webserver started at http://127.0.98.1:33519/ using document root <none> and password file <none>
I20260927 16:42:38.265492 17521 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:38.265583 17521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:38.265810 17521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:38.268126 17521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-0/data/instance:
uuid: "4336a1e6207b4ee08d4782e860a51c20"
format_stamp: "Formatted at 2026-09-27 16:42:38 on dist-test-slave-z40b"
server_key: "6beea2da03005e4c61709bfc932aa655"
server_key_iv: "2c317fdbb0aee8af1fe55e3898d5a921"
server_key_version: "encryptionkey@0"
I20260927 16:42:38.268795 17521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4336a1e6207b4ee08d4782e860a51c20"
format_stamp: "Formatted at 2026-09-27 16:42:38 on dist-test-slave-z40b"
server_key: "6beea2da03005e4c61709bfc932aa655"
server_key_iv: "2c317fdbb0aee8af1fe55e3898d5a921"
server_key_version: "encryptionkey@0"
I20260927 16:42:38.272416 17521 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:42:38.274646 17536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:38.275808 17521 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:42:38.275940 17521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "4336a1e6207b4ee08d4782e860a51c20"
format_stamp: "Formatted at 2026-09-27 16:42:38 on dist-test-slave-z40b"
server_key: "6beea2da03005e4c61709bfc932aa655"
server_key_iv: "2c317fdbb0aee8af1fe55e3898d5a921"
server_key_version: "encryptionkey@0"
I20260927 16:42:38.276058 17521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:38.292404 17521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:38.295352 17521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:38.295547 17521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:38.296108 17521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:42:38.296905 17521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:42:38.296977 17521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:38.297050 17521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:42:38.297102 17521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:38.305707 17521 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35087
I20260927 16:42:38.305791 17649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35087 every 8 connection(s)
I20260927 16:42:38.306612 17521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 16:42:38 dist-test-slave-z40b krb5kdc[17414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:42:38.315963 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 17521
I20260927 16:42:38.316105 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:42:38.316407 392 external_mini_cluster.cc:1428] Setting key 41c488f0292a74664b5ab1d6b9008c7f
I20260927 16:42:38.320796 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:38.308455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47363 (local address 127.0.98.62:36865)
0927 16:42:38.308604 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:38.308608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:38.309414 (+ 806us) server_negotiation.cc:407] Connection header received
0927 16:42:38.310298 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:38.310301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:38.310360 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:38.310467 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:38.311657 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.312225 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.313378 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.313706 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.313815 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:38.316722 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:38.316726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:38.316728 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:38.316759 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:38.318411 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:38.319047 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:38.319049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:38.319050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:38.319123 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:38.319566 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:38.319571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:38.319574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:38.319820 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:38.319928 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:38.320416 (+ 488us) server_negotiation.cc:299] Negotiation successful
0927 16:42:38.320555 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 16:42:38.321543 17652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:38.308810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36865 (local address 127.0.98.1:47363)
0927 16:42:38.309255 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:38.309297 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:38.310094 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:38.310651 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:38.310658 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:38.311025 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:42:38.311506 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:38.311519 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.312376 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.312381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:38.313248 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:38.313258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.313861 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.313865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:38.314065 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:38.314748 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:42:38.314767 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:42:38.316555 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:42:38.318536 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:38.318545 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:38.318564 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:38.318940 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:38.319227 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:38.319233 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:38.319238 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:38.319454 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:38.319931 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:42:38.319935 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:42:38.320169 (+ 234us) client_negotiation.cc:769] Sending connection context
0927 16:42:38.320321 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 16:42:38.320589 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":123,"threads_started":1}
I20260927 16:42:38.322817 17650 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36865
I20260927 16:42:38.323099 17650 heartbeater.cc:464] Registering TS with master...
I20260927 16:42:38.323755 17650 heartbeater.cc:511] Master 127.0.98.62:36865 requested a full tablet report, sending...
I20260927 16:42:38.325357 17459 ts_manager.cc:194] Registered new tserver with Master: 4336a1e6207b4ee08d4782e860a51c20 (127.0.98.1:35087)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:42:38.326522 17459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:42:38.342626 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36865
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:42:38.447151 17655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:38.447432 17655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:38.447504 17655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:38.451339 17655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:38.451413 17655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:38.451493 17655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:42:38.456611 17655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:38.457517 17655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:38.458494 17655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:38.467604 17660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:38.467607 17663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:38.467604 17661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:38.467988 17655 server_base.cc:1034] running on GCE node
I20260927 16:42:38.468412 17655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:38.469115 17655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:38.470271 17655 hybrid_clock.cc:648] HybridClock initialized: now 1790527358470259 us; error 23 us; skew 500 ppm
Sep 27 16:42:38 dist-test-slave-z40b krb5kdc[17414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:38.472994 17655 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:42:38.473791 17655 webserver.cc:460] Webserver started at http://127.0.98.2:41637/ using document root <none> and password file <none>
I20260927 16:42:38.474363 17655 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:38.474452 17655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:38.474658 17655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:38.476622 17655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-1/data/instance:
uuid: "50153b166407461fb435e6cb6078c3f7"
format_stamp: "Formatted at 2026-09-27 16:42:38 on dist-test-slave-z40b"
server_key: "c09b576437b3597d1883edbdec803f33"
server_key_iv: "524e4dd73d3d24e3c53ec79b5f147432"
server_key_version: "encryptionkey@0"
I20260927 16:42:38.477156 17655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-1/wal/instance:
uuid: "50153b166407461fb435e6cb6078c3f7"
format_stamp: "Formatted at 2026-09-27 16:42:38 on dist-test-slave-z40b"
server_key: "c09b576437b3597d1883edbdec803f33"
server_key_iv: "524e4dd73d3d24e3c53ec79b5f147432"
server_key_version: "encryptionkey@0"
I20260927 16:42:38.480754 17655 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 16:42:38.482962 17670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:38.484122 17655 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:42:38.484241 17655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "50153b166407461fb435e6cb6078c3f7"
format_stamp: "Formatted at 2026-09-27 16:42:38 on dist-test-slave-z40b"
server_key: "c09b576437b3597d1883edbdec803f33"
server_key_iv: "524e4dd73d3d24e3c53ec79b5f147432"
server_key_version: "encryptionkey@0"
I20260927 16:42:38.484337 17655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:38.502750 17655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:38.505798 17655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:38.506006 17655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:38.506523 17655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:42:38.507304 17655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:42:38.507369 17655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:38.507436 17655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:42:38.507476 17655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:38.516134 17655 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41115
I20260927 16:42:38.516160 17783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41115 every 8 connection(s)
I20260927 16:42:38.517007 17655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:42:38.518062 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 17655
I20260927 16:42:38.518193 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:42:38.518492 392 external_mini_cluster.cc:1428] Setting key eab17d4e1d99735732a9c797c6aa1519
Sep 27 16:42:38 dist-test-slave-z40b krb5kdc[17414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 16:42:38.530961 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:38.518737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60523 (local address 127.0.98.62:36865)
0927 16:42:38.518879 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:38.518884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:38.520106 (+ 1222us) server_negotiation.cc:407] Connection header received
0927 16:42:38.520927 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:38.520933 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:38.520994 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:38.521082 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:38.522539 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.523033 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.524765 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.524965 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.525041 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:38.527723 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:38.527727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:38.527730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:38.527760 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:38.529071 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:38.529591 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:38.529594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:38.529595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:38.529662 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:38.529932 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:38.529934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:38.529935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:38.530085 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:38.530152 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:38.530601 (+ 449us) server_negotiation.cc:299] Negotiation successful
0927 16:42:38.530725 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:42:38.531776 17786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:38.519194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36865 (local address 127.0.98.2:60523)
0927 16:42:38.519940 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:38.520003 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:38.520737 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:38.521234 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:38.521240 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:38.521714 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:42:38.522384 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:38.522404 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.523158 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.523164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:38.524643 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:38.524657 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.525116 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.525123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:38.525351 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:38.525910 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:42:38.525932 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:42:38.527555 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:42:38.529207 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:38.529216 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:38.529232 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:38.529494 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:38.529736 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:38.529737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:38.529739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:38.529853 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:38.530179 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:42:38.530184 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:42:38.530382 (+ 198us) client_negotiation.cc:769] Sending connection context
0927 16:42:38.530532 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 16:42:38.530878 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":104,"threads_started":1}
I20260927 16:42:38.532958 17784 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36865
I20260927 16:42:38.533195 17784 heartbeater.cc:464] Registering TS with master...
I20260927 16:42:38.533628 17784 heartbeater.cc:511] Master 127.0.98.62:36865 requested a full tablet report, sending...
I20260927 16:42:38.534657 17459 ts_manager.cc:194] Registered new tserver with Master: 50153b166407461fb435e6cb6078c3f7 (127.0.98.2:41115)
I20260927 16:42:38.535179 17459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:42:38.545382 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36865
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:42:38.650631 17789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:38.650838 17789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:38.650873 17789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:38.654547 17789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:38.654593 17789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:38.654644 17789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:42:38.659435 17789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:38.660245 17789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:38.661092 17789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:38.670248 17794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:38.670282 17795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:38.670307 17797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:38.670696 17789 server_base.cc:1034] running on GCE node
I20260927 16:42:38.671178 17789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:38.671832 17789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:38.673012 17789 hybrid_clock.cc:648] HybridClock initialized: now 1790527358672994 us; error 27 us; skew 500 ppm
Sep 27 16:42:38 dist-test-slave-z40b krb5kdc[17414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:38.675714 17789 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:42:38.676473 17789 webserver.cc:460] Webserver started at http://127.0.98.3:36347/ using document root <none> and password file <none>
I20260927 16:42:38.676963 17789 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:38.677016 17789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:38.677170 17789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:38.679098 17789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-2/data/instance:
uuid: "e9a3afc1be374b79a5ce9d0683794921"
format_stamp: "Formatted at 2026-09-27 16:42:38 on dist-test-slave-z40b"
server_key: "2a8f5ca23af6f3b80ac15a416cc8c42a"
server_key_iv: "179cad989d14d6d8189789863ff72fae"
server_key_version: "encryptionkey@0"
I20260927 16:42:38.679646 17789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e9a3afc1be374b79a5ce9d0683794921"
format_stamp: "Formatted at 2026-09-27 16:42:38 on dist-test-slave-z40b"
server_key: "2a8f5ca23af6f3b80ac15a416cc8c42a"
server_key_iv: "179cad989d14d6d8189789863ff72fae"
server_key_version: "encryptionkey@0"
I20260927 16:42:38.683059 17789 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:42:38.685269 17804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:38.686401 17789 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:42:38.686523 17789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "e9a3afc1be374b79a5ce9d0683794921"
format_stamp: "Formatted at 2026-09-27 16:42:38 on dist-test-slave-z40b"
server_key: "2a8f5ca23af6f3b80ac15a416cc8c42a"
server_key_iv: "179cad989d14d6d8189789863ff72fae"
server_key_version: "encryptionkey@0"
I20260927 16:42:38.686619 17789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:38.703382 17789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:38.705905 17789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:38.706097 17789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:38.706638 17789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:42:38.707505 17789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:42:38.707551 17789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:38.707613 17789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:42:38.707644 17789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:38.716619 17789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41377
I20260927 16:42:38.716645 17917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41377 every 8 connection(s)
I20260927 16:42:38.717459 17789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:42:38.720709 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 17789
I20260927 16:42:38.720832 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:42:38.721089 392 external_mini_cluster.cc:1428] Setting key 00a5768810dcd99220eb706b46e2ee00
Sep 27 16:42:38 dist-test-slave-z40b krb5kdc[17414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:42:38.730057 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:38.719138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42775 (local address 127.0.98.62:36865)
0927 16:42:38.719318 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:38.719323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:38.720352 (+ 1029us) server_negotiation.cc:407] Connection header received
0927 16:42:38.721320 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:38.721324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:38.721376 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:38.721461 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:38.722747 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.723393 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.724453 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.724645 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.724712 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:38.727050 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:38.727053 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:38.727055 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:38.727077 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:38.728426 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:38.728884 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:38.728887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:38.728888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:38.728952 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:38.729221 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:38.729223 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:38.729225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:38.729385 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:38.729455 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:38.729858 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 16:42:38.729941 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 16:42:38.730759 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:38.719539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36865 (local address 127.0.98.3:42775)
0927 16:42:38.720197 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:38.720237 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:38.721065 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:38.721651 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:38.721658 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:38.722090 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:42:38.722627 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:38.722639 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.723540 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.723542 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:38.724359 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:38.724368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.724753 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.724755 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:38.724910 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:38.725473 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:42:38.725487 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:42:38.726906 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:42:38.728532 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:38.728536 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:38.728547 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:38.728787 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:38.729026 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:38.729028 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:38.729029 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:38.729148 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:38.729467 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:42:38.729471 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:42:38.729665 (+ 194us) client_negotiation.cc:769] Sending connection context
0927 16:42:38.729805 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 16:42:38.730019 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":132,"threads_started":1}
I20260927 16:42:38.731860 17918 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36865
I20260927 16:42:38.732080 17918 heartbeater.cc:464] Registering TS with master...
I20260927 16:42:38.732523 17918 heartbeater.cc:511] Master 127.0.98.62:36865 requested a full tablet report, sending...
I20260927 16:42:38.733807 17459 ts_manager.cc:194] Registered new tserver with Master: e9a3afc1be374b79a5ce9d0683794921 (127.0.98.3:41377)
I20260927 16:42:38.734488 17459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42775
I20260927 16:42:38.734858 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:42:38.743347 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:38.736831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40090 (local address 127.0.98.62:36865)
0927 16:42:38.736943 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:38.736946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:38.737118 (+ 172us) server_negotiation.cc:407] Connection header received
0927 16:42:38.737196 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:38.737198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:38.737244 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:38.737305 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:38.737884 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.738370 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.739159 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.739465 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.739546 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:38.740679 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:38.740682 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:38.740684 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:38.740704 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:38.742072 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:38.742380 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:38.742382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:38.742383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:38.742456 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:38.742669 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:38.742671 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:38.742672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:38.742844 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:38.742936 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:38.743094 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 16:42:38.743170 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:42:38.745800 17459 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:42:38.748776 17459 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:42:38.762193 17931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:38.756536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54312 (local address 127.0.98.2:41115)
0927 16:42:38.757683 (+ 1147us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:38.757698 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:38.757721 (+ 23us) server_negotiation.cc:407] Connection header received
0927 16:42:38.757794 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:38.757799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:38.757993 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:38.758417 (+ 424us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:38.758573 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.759349 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.760876 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.761543 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.761660 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:38.761795 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:38.761939 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 16:42:38.762020 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":886,"thread_start_us":152,"threads_started":1}
I20260927 16:42:38.762185 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:38.756379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41115 (local address 127.0.0.1:54312)
0927 16:42:38.757066 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:38.757116 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:38.757254 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:38.758151 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:38.758154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:38.758172 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:38.758277 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:38.758284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.759474 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.759486 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:38.760770 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:38.760780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.761661 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.761665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:38.761763 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:38.761795 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 16:42:38.761902 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 16:42:38.762021 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":28,"spinlock_wait_cycles":7040,"thread_start_us":64,"threads_started":1}
I20260927 16:42:38.762496 17929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:38.756469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35087 (local address 127.0.0.1:33178)
0927 16:42:38.757066 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:38.757116 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:38.757254 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:38.758105 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:38.758111 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:38.758134 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:38.758258 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:38.758263 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.759862 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.759866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:38.761021 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:38.761027 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.762103 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.762107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:38.762189 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:38.762210 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 16:42:38.762261 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 16:42:38.762330 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":60,"spinlock_wait_cycles":8576,"thread_start_us":82,"threads_started":1}
I20260927 16:42:38.762835 17933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:38.757126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41064 (local address 127.0.98.3:41377)
0927 16:42:38.757835 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:38.757845 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:38.757862 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:42:38.757919 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:38.757922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:38.758065 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:38.758194 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:38.758540 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.759724 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.761248 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.761947 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.762081 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:38.762212 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:38.762576 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 16:42:38.762673 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":131,"threads_started":1}
I20260927 16:42:38.763525 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:38.757141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33178 (local address 127.0.98.1:35087)
0927 16:42:38.757570 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:38.757582 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:38.757611 (+ 29us) server_negotiation.cc:407] Connection header received
0927 16:42:38.757717 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:38.757723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:38.757926 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:38.758376 (+ 450us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:38.758529 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.759728 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.761131 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.761947 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.762595 (+ 648us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:38.763218 (+ 623us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:38.763286 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 16:42:38.763372 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":120,"threads_started":1}
I20260927 16:42:38.762714 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:38.756730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41377 (local address 127.0.0.1:41064)
0927 16:42:38.757497 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:38.757510 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:38.757588 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:38.758336 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:38.758339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:38.758358 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:38.758425 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:38.758430 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.759894 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.759897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:38.761119 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:38.761129 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.762270 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.762273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:38.762436 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:38.762455 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 16:42:38.762532 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 16:42:38.762597 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":60,"thread_start_us":79,"threads_started":1}
I20260927 16:42:38.765763 17718 tablet_service.cc:1467] Processing CreateTablet for tablet 9d1fd2db616a456cac382377f3ca252b (DEFAULT_TABLE table=test-table [id=48eedb15e9d94e8a991f655279e8a6c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:42:38.766141 17852 tablet_service.cc:1467] Processing CreateTablet for tablet 9d1fd2db616a456cac382377f3ca252b (DEFAULT_TABLE table=test-table [id=48eedb15e9d94e8a991f655279e8a6c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:42:38.766960 17718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d1fd2db616a456cac382377f3ca252b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:38.767092 17852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d1fd2db616a456cac382377f3ca252b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:38.768385 17584 tablet_service.cc:1467] Processing CreateTablet for tablet 9d1fd2db616a456cac382377f3ca252b (DEFAULT_TABLE table=test-table [id=48eedb15e9d94e8a991f655279e8a6c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:42:38.769335 17584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d1fd2db616a456cac382377f3ca252b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:38.774211 17935 tablet_bootstrap.cc:492] T 9d1fd2db616a456cac382377f3ca252b P 50153b166407461fb435e6cb6078c3f7: Bootstrap starting.
I20260927 16:42:38.774809 17936 tablet_bootstrap.cc:492] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921: Bootstrap starting.
I20260927 16:42:38.775514 17937 tablet_bootstrap.cc:492] T 9d1fd2db616a456cac382377f3ca252b P 4336a1e6207b4ee08d4782e860a51c20: Bootstrap starting.
I20260927 16:42:38.777057 17936 tablet_bootstrap.cc:654] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:38.777311 17937 tablet_bootstrap.cc:654] T 9d1fd2db616a456cac382377f3ca252b P 4336a1e6207b4ee08d4782e860a51c20: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:38.777611 17935 tablet_bootstrap.cc:654] T 9d1fd2db616a456cac382377f3ca252b P 50153b166407461fb435e6cb6078c3f7: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:38.778439 17936 log.cc:824] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:38.778508 17937 log.cc:824] T 9d1fd2db616a456cac382377f3ca252b P 4336a1e6207b4ee08d4782e860a51c20: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:38.778719 17935 log.cc:824] T 9d1fd2db616a456cac382377f3ca252b P 50153b166407461fb435e6cb6078c3f7: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:38.780674 17937 tablet_bootstrap.cc:492] T 9d1fd2db616a456cac382377f3ca252b P 4336a1e6207b4ee08d4782e860a51c20: No bootstrap required, opened a new log
I20260927 16:42:38.780680 17936 tablet_bootstrap.cc:492] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921: No bootstrap required, opened a new log
I20260927 16:42:38.780859 17936 ts_tablet_manager.cc:1397] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:42:38.780859 17937 ts_tablet_manager.cc:1397] T 9d1fd2db616a456cac382377f3ca252b P 4336a1e6207b4ee08d4782e860a51c20: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:42:38.784192 17935 tablet_bootstrap.cc:492] T 9d1fd2db616a456cac382377f3ca252b P 50153b166407461fb435e6cb6078c3f7: No bootstrap required, opened a new log
I20260927 16:42:38.784415 17935 ts_tablet_manager.cc:1397] T 9d1fd2db616a456cac382377f3ca252b P 50153b166407461fb435e6cb6078c3f7: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260927 16:42:38.784694 17936 raft_consensus.cc:348] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50153b166407461fb435e6cb6078c3f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41115 } } peers { permanent_uuid: "4336a1e6207b4ee08d4782e860a51c20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35087 } } peers { permanent_uuid: "e9a3afc1be374b79a5ce9d0683794921" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41377 } }
I20260927 16:42:38.784694 17937 raft_consensus.cc:348] T 9d1fd2db616a456cac382377f3ca252b P 4336a1e6207b4ee08d4782e860a51c20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50153b166407461fb435e6cb6078c3f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41115 } } peers { permanent_uuid: "4336a1e6207b4ee08d4782e860a51c20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35087 } } peers { permanent_uuid: "e9a3afc1be374b79a5ce9d0683794921" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41377 } }
I20260927 16:42:38.784935 17937 raft_consensus.cc:374] T 9d1fd2db616a456cac382377f3ca252b P 4336a1e6207b4ee08d4782e860a51c20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:38.784935 17936 raft_consensus.cc:374] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:38.784982 17937 raft_consensus.cc:729] T 9d1fd2db616a456cac382377f3ca252b P 4336a1e6207b4ee08d4782e860a51c20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4336a1e6207b4ee08d4782e860a51c20, State: Initialized, Role: FOLLOWER
I20260927 16:42:38.784991 17936 raft_consensus.cc:729] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a3afc1be374b79a5ce9d0683794921, State: Initialized, Role: FOLLOWER
I20260927 16:42:38.785423 17936 consensus_queue.cc:260] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50153b166407461fb435e6cb6078c3f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41115 } } peers { permanent_uuid: "4336a1e6207b4ee08d4782e860a51c20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35087 } } peers { permanent_uuid: "e9a3afc1be374b79a5ce9d0683794921" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41377 } }
I20260927 16:42:38.785423 17937 consensus_queue.cc:260] T 9d1fd2db616a456cac382377f3ca252b P 4336a1e6207b4ee08d4782e860a51c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50153b166407461fb435e6cb6078c3f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41115 } } peers { permanent_uuid: "4336a1e6207b4ee08d4782e860a51c20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35087 } } peers { permanent_uuid: "e9a3afc1be374b79a5ce9d0683794921" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41377 } }
I20260927 16:42:38.786291 17650 heartbeater.cc:503] Master 127.0.98.62:36865 was elected leader, sending a full tablet report...
I20260927 16:42:38.786334 17918 heartbeater.cc:503] Master 127.0.98.62:36865 was elected leader, sending a full tablet report...
I20260927 16:42:38.786414 17937 ts_tablet_manager.cc:1428] T 9d1fd2db616a456cac382377f3ca252b P 4336a1e6207b4ee08d4782e860a51c20: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260927 16:42:38.787631 17936 ts_tablet_manager.cc:1428] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 16:42:38.788270 17935 raft_consensus.cc:348] T 9d1fd2db616a456cac382377f3ca252b P 50153b166407461fb435e6cb6078c3f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50153b166407461fb435e6cb6078c3f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41115 } } peers { permanent_uuid: "4336a1e6207b4ee08d4782e860a51c20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35087 } } peers { permanent_uuid: "e9a3afc1be374b79a5ce9d0683794921" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41377 } }
I20260927 16:42:38.788547 17935 raft_consensus.cc:374] T 9d1fd2db616a456cac382377f3ca252b P 50153b166407461fb435e6cb6078c3f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:38.788605 17935 raft_consensus.cc:729] T 9d1fd2db616a456cac382377f3ca252b P 50153b166407461fb435e6cb6078c3f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50153b166407461fb435e6cb6078c3f7, State: Initialized, Role: FOLLOWER
I20260927 16:42:38.789045 17935 consensus_queue.cc:260] T 9d1fd2db616a456cac382377f3ca252b P 50153b166407461fb435e6cb6078c3f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50153b166407461fb435e6cb6078c3f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41115 } } peers { permanent_uuid: "4336a1e6207b4ee08d4782e860a51c20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35087 } } peers { permanent_uuid: "e9a3afc1be374b79a5ce9d0683794921" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41377 } }
I20260927 16:42:38.789646 17784 heartbeater.cc:503] Master 127.0.98.62:36865 was elected leader, sending a full tablet report...
I20260927 16:42:38.790028 17935 ts_tablet_manager.cc:1428] T 9d1fd2db616a456cac382377f3ca252b P 50153b166407461fb435e6cb6078c3f7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 16:42:38.807771 17651 tablet.cc:2369] T 9d1fd2db616a456cac382377f3ca252b P 4336a1e6207b4ee08d4782e860a51c20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:42:38.822181 17941 raft_consensus.cc:482] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:42:38.822403 17941 raft_consensus.cc:504] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50153b166407461fb435e6cb6078c3f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41115 } } peers { permanent_uuid: "4336a1e6207b4ee08d4782e860a51c20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35087 } } peers { permanent_uuid: "e9a3afc1be374b79a5ce9d0683794921" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41377 } }
I20260927 16:42:38.823607 17941 leader_election.cc:288] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50153b166407461fb435e6cb6078c3f7 (127.0.98.2:41115), 4336a1e6207b4ee08d4782e860a51c20 (127.0.98.1:35087)
I20260927 16:42:38.828310 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:38.823838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35087 (local address 127.0.98.3:33883)
0927 16:42:38.824022 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:38.824037 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:38.824123 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:38.824496 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:38.824501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:38.824526 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:38.824617 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:38.824621 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.825623 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.825629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:38.826993 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:38.827006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.827873 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.827876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:38.827989 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:38.828014 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 16:42:38.828094 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 16:42:38.828164 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260927 16:42:38.828328 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:38.823929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33883 (local address 127.0.98.1:35087)
0927 16:42:38.824057 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:38.824064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:38.824080 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:42:38.824249 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:38.824252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:38.824302 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:38.824376 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:38.824731 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.825521 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.827150 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.827752 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.827865 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:38.827950 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:38.828125 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 16:42:38.828199 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 16:42:38.828886 17931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:38.824405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42991 (local address 127.0.98.2:41115)
0927 16:42:38.824613 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:38.824619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:38.824634 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:42:38.824721 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:38.824726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:38.824806 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:38.824914 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:38.825280 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.826100 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.827610 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.828158 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.828249 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:38.828303 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:38.828676 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 16:42:38.828757 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 16:42:38.829252 17944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:38.823944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41115 (local address 127.0.98.3:42991)
0927 16:42:38.824334 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:38.824348 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:38.824419 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:38.824923 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:38.824961 (+ 38us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:38.824983 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:38.825113 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:38.825119 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.826240 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.826246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:38.827485 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:38.827502 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.828377 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.828383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:38.828510 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:38.828540 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 16:42:38.829052 (+ 512us) client_negotiation.cc:240] Negotiation successful
0927 16:42:38.829137 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":186,"threads_started":1}
I20260927 16:42:38.829306 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d1fd2db616a456cac382377f3ca252b" candidate_uuid: "e9a3afc1be374b79a5ce9d0683794921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4336a1e6207b4ee08d4782e860a51c20" is_pre_election: true
I20260927 16:42:38.829635 17604 raft_consensus.cc:2454] T 9d1fd2db616a456cac382377f3ca252b P 4336a1e6207b4ee08d4782e860a51c20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9a3afc1be374b79a5ce9d0683794921 in term 0.
I20260927 16:42:38.829927 17738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d1fd2db616a456cac382377f3ca252b" candidate_uuid: "e9a3afc1be374b79a5ce9d0683794921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50153b166407461fb435e6cb6078c3f7" is_pre_election: true
I20260927 16:42:38.830137 17808 leader_election.cc:302] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4336a1e6207b4ee08d4782e860a51c20, e9a3afc1be374b79a5ce9d0683794921; no voters:
I20260927 16:42:38.830209 17738 raft_consensus.cc:2454] T 9d1fd2db616a456cac382377f3ca252b P 50153b166407461fb435e6cb6078c3f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9a3afc1be374b79a5ce9d0683794921 in term 0.
I20260927 16:42:38.830390 17941 raft_consensus.cc:2781] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:42:38.830502 17941 raft_consensus.cc:482] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:42:38.830564 17941 raft_consensus.cc:3037] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:38.831838 17941 raft_consensus.cc:504] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50153b166407461fb435e6cb6078c3f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41115 } } peers { permanent_uuid: "4336a1e6207b4ee08d4782e860a51c20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35087 } } peers { permanent_uuid: "e9a3afc1be374b79a5ce9d0683794921" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41377 } }
I20260927 16:42:38.832183 17941 leader_election.cc:288] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921 [CANDIDATE]: Term 1 election: Requested vote from peers 50153b166407461fb435e6cb6078c3f7 (127.0.98.2:41115), 4336a1e6207b4ee08d4782e860a51c20 (127.0.98.1:35087)
I20260927 16:42:38.832479 17738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d1fd2db616a456cac382377f3ca252b" candidate_uuid: "e9a3afc1be374b79a5ce9d0683794921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50153b166407461fb435e6cb6078c3f7"
I20260927 16:42:38.832552 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d1fd2db616a456cac382377f3ca252b" candidate_uuid: "e9a3afc1be374b79a5ce9d0683794921" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4336a1e6207b4ee08d4782e860a51c20"
I20260927 16:42:38.832597 17738 raft_consensus.cc:3037] T 9d1fd2db616a456cac382377f3ca252b P 50153b166407461fb435e6cb6078c3f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:38.832685 17604 raft_consensus.cc:3037] T 9d1fd2db616a456cac382377f3ca252b P 4336a1e6207b4ee08d4782e860a51c20 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:38.833629 17738 raft_consensus.cc:2454] T 9d1fd2db616a456cac382377f3ca252b P 50153b166407461fb435e6cb6078c3f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9a3afc1be374b79a5ce9d0683794921 in term 1.
I20260927 16:42:38.833635 17604 raft_consensus.cc:2454] T 9d1fd2db616a456cac382377f3ca252b P 4336a1e6207b4ee08d4782e860a51c20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9a3afc1be374b79a5ce9d0683794921 in term 1.
I20260927 16:42:38.833954 17807 leader_election.cc:302] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50153b166407461fb435e6cb6078c3f7, e9a3afc1be374b79a5ce9d0683794921; no voters:
I20260927 16:42:38.834162 17941 raft_consensus.cc:2781] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:42:38.834419 17941 raft_consensus.cc:686] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921 [term 1 LEADER]: Becoming Leader. State: Replica: e9a3afc1be374b79a5ce9d0683794921, State: Running, Role: LEADER
I20260927 16:42:38.834738 17941 consensus_queue.cc:237] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50153b166407461fb435e6cb6078c3f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41115 } } peers { permanent_uuid: "4336a1e6207b4ee08d4782e860a51c20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35087 } } peers { permanent_uuid: "e9a3afc1be374b79a5ce9d0683794921" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41377 } }
I20260927 16:42:38.837867 17459 catalog_manager.cc:5495] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9a3afc1be374b79a5ce9d0683794921 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e9a3afc1be374b79a5ce9d0683794921" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50153b166407461fb435e6cb6078c3f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4336a1e6207b4ee08d4782e860a51c20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9a3afc1be374b79a5ce9d0683794921" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41377 } health_report { overall_health: HEALTHY } } }
I20260927 16:42:38.858160 17933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:38.854905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41076 (local address 127.0.98.3:41377)
0927 16:42:38.855027 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:38.855030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:38.855044 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:42:38.855180 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:38.855182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:38.855260 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:38.855346 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:42:38.855655 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.856151 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.856944 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:38.857146 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:38.857218 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:38.857472 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:42:38.857862 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:42:38.857965 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:38.858017 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 16:42:38.858061 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:42:38.863734 17604 raft_consensus.cc:1261] T 9d1fd2db616a456cac382377f3ca252b P 4336a1e6207b4ee08d4782e860a51c20 [term 1 FOLLOWER]: Refusing update from remote peer e9a3afc1be374b79a5ce9d0683794921: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:42:38.863734 17738 raft_consensus.cc:1261] T 9d1fd2db616a456cac382377f3ca252b P 50153b166407461fb435e6cb6078c3f7 [term 1 FOLLOWER]: Refusing update from remote peer e9a3afc1be374b79a5ce9d0683794921: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:42:38.864459 17945 consensus_queue.cc:1035] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4336a1e6207b4ee08d4782e860a51c20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:42:38.864639 17941 consensus_queue.cc:1035] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50153b166407461fb435e6cb6078c3f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:42:38.874819 17950 mvcc.cc:205] Tried to move back new op lower bound from 7334000061899026432 to 7334000061790248960. Current Snapshot: MvccSnapshot[applied={T|T < 7334000061899026432}]
I20260927 16:42:38.879563 17458 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:42:38.879727 17458 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:42:38.881927 17458 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1dd72b9f75142acb77abbac596b5154: Sending DeleteTablet for 3 replicas of tablet 9d1fd2db616a456cac382377f3ca252b
I20260927 16:42:38.882627 17584 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d1fd2db616a456cac382377f3ca252b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:42:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33178
I20260927 16:42:38.882648 17718 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d1fd2db616a456cac382377f3ca252b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:42:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54312
I20260927 16:42:38.883087 17957 tablet_replica.cc:330] T 9d1fd2db616a456cac382377f3ca252b P 50153b166407461fb435e6cb6078c3f7: stopping tablet replica
I20260927 16:42:38.883077 17852 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d1fd2db616a456cac382377f3ca252b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:42:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41064
I20260927 16:42:38.883466 17957 raft_consensus.cc:2229] T 9d1fd2db616a456cac382377f3ca252b P 50153b166407461fb435e6cb6078c3f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:42:38.883504 17958 tablet_replica.cc:330] T 9d1fd2db616a456cac382377f3ca252b P 4336a1e6207b4ee08d4782e860a51c20: stopping tablet replica
I20260927 16:42:38.883754 17958 raft_consensus.cc:2229] T 9d1fd2db616a456cac382377f3ca252b P 4336a1e6207b4ee08d4782e860a51c20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:42:38.883826 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 17521
I20260927 16:42:38.883807 17957 raft_consensus.cc:2258] T 9d1fd2db616a456cac382377f3ca252b P 50153b166407461fb435e6cb6078c3f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:42:38.885778 17957 ts_tablet_manager.cc:1905] T 9d1fd2db616a456cac382377f3ca252b P 50153b166407461fb435e6cb6078c3f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:42:38.887363 17959 tablet_replica.cc:330] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921: stopping tablet replica
I20260927 16:42:38.887718 17959 raft_consensus.cc:2229] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:42:38.888140 17959 raft_consensus.cc:2258] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:42:38.889345 17959 ts_tablet_manager.cc:1905] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:42:38.889549 17957 ts_tablet_manager.cc:1918] T 9d1fd2db616a456cac382377f3ca252b P 50153b166407461fb435e6cb6078c3f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:42:38.889647 17957 log.cc:1196] T 9d1fd2db616a456cac382377f3ca252b P 50153b166407461fb435e6cb6078c3f7: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-1/wal/wals/9d1fd2db616a456cac382377f3ca252b
I20260927 16:42:38.890038 17957 ts_tablet_manager.cc:1939] T 9d1fd2db616a456cac382377f3ca252b P 50153b166407461fb435e6cb6078c3f7: Deleting consensus metadata
W20260927 16:42:38.891129 17446 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35087 (error 108)
I20260927 16:42:38.891507 17445 catalog_manager.cc:4841] TS 50153b166407461fb435e6cb6078c3f7 (127.0.98.2:41115): tablet 9d1fd2db616a456cac382377f3ca252b (table test-table [id=48eedb15e9d94e8a991f655279e8a6c8]) successfully deleted
I20260927 16:42:38.891950 17959 ts_tablet_manager.cc:1918] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:42:38.891961 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 17655
I20260927 16:42:38.892274 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:38.891952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35087 (local address 127.0.0.1:33188)
0927 16:42:38.892107 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:38.892182 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 16:42:38.892881 17446 catalog_manager.cc:4584] TS 4336a1e6207b4ee08d4782e860a51c20 (127.0.98.1:35087): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9d1fd2db616a456cac382377f3ca252b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35087: connect: Connection refused (error 111)
I20260927 16:42:38.896829 17959 log.cc:1196] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790527121515808-392-0/minicluster-data/ts-2/wal/wals/9d1fd2db616a456cac382377f3ca252b
I20260927 16:42:38.897184 17959 ts_tablet_manager.cc:1939] T 9d1fd2db616a456cac382377f3ca252b P e9a3afc1be374b79a5ce9d0683794921: Deleting consensus metadata
I20260927 16:42:38.897348 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 17789
I20260927 16:42:38.901831 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 17427
2026-09-27T16:42:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2168 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T16:42:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:42:38Z Disabled control of system clock
I20260927 16:42:38.929821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35213
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 16:42:39.034286 17964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:39.034529 17964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:39.034580 17964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:39.038586 17964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:42:39.038636 17964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:39.038651 17964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:39.038666 17964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:42:39.038681 17964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:42:39.043115 17964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35213
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:39.044149 17964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:39.045396 17964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:39.054867 17969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:39.054867 17970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:39.054960 17972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:39.055336 17964 server_base.cc:1034] running on GCE node
I20260927 16:42:39.056299 17964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:39.057986 17964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:39.059336 17964 hybrid_clock.cc:648] HybridClock initialized: now 1790527359059257 us; error 51 us; skew 500 ppm
I20260927 16:42:39.061877 17964 webserver.cc:460] Webserver started at http://127.0.98.62:40039/ using document root <none> and password file <none>
I20260927 16:42:39.062908 17964 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:39.063020 17964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:39.063267 17964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:39.066636 17964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "4867ec7e4040481399d36b1bf41b934a"
format_stamp: "Formatted at 2026-09-27 16:42:39 on dist-test-slave-z40b"
server_key: "28eca3cfb4032d31c69619eb5c7e07f1"
server_key_iv: "05ffb58be109deb220892810b8271221"
server_key_version: "encryptionkey@0"
I20260927 16:42:39.067308 17964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "4867ec7e4040481399d36b1bf41b934a"
format_stamp: "Formatted at 2026-09-27 16:42:39 on dist-test-slave-z40b"
server_key: "28eca3cfb4032d31c69619eb5c7e07f1"
server_key_iv: "05ffb58be109deb220892810b8271221"
server_key_version: "encryptionkey@0"
I20260927 16:42:39.072235 17964 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 16:42:39.075404 17978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:39.076928 17964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 16:42:39.077070 17964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "4867ec7e4040481399d36b1bf41b934a"
format_stamp: "Formatted at 2026-09-27 16:42:39 on dist-test-slave-z40b"
server_key: "28eca3cfb4032d31c69619eb5c7e07f1"
server_key_iv: "05ffb58be109deb220892810b8271221"
server_key_version: "encryptionkey@0"
I20260927 16:42:39.077193 17964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:39.104269 17964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:39.104893 17964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:39.105087 17964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:39.111831 17964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35213
I20260927 16:42:39.111863 18030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35213 every 8 connection(s)
I20260927 16:42:39.112692 17964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:42:39.114974 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 17964
I20260927 16:42:39.115093 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:42:39.115435 392 external_mini_cluster.cc:1428] Setting key 02c689e59e29071becbc33c176542ddb
I20260927 16:42:39.115828 18031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:39.122423 18031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4867ec7e4040481399d36b1bf41b934a: Bootstrap starting.
I20260927 16:42:39.124708 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:39.116812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49672 (local address 127.0.98.62:35213)
0927 16:42:39.117395 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:39.117410 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:39.117444 (+ 34us) server_negotiation.cc:407] Connection header received
0927 16:42:39.118449 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:39.118467 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:39.118835 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:39.119119 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:39.119505 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.120425 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.121591 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.122024 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.122128 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:39.123047 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:39.123052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:39.123063 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:42:39.123481 (+ 418us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:39.123533 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:42:39.123552 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:39.123626 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:39.123785 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 16:42:39.124066 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":133,"threads_started":1}
I20260927 16:42:39.124797 18031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4867ec7e4040481399d36b1bf41b934a: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:39.125674 18031 log.cc:824] T 00000000000000000000000000000000 P 4867ec7e4040481399d36b1bf41b934a: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:39.127655 18031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4867ec7e4040481399d36b1bf41b934a: No bootstrap required, opened a new log
I20260927 16:42:39.131016 18031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4867ec7e4040481399d36b1bf41b934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4867ec7e4040481399d36b1bf41b934a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35213 } }
I20260927 16:42:39.131248 18031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4867ec7e4040481399d36b1bf41b934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:39.131330 18031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4867ec7e4040481399d36b1bf41b934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4867ec7e4040481399d36b1bf41b934a, State: Initialized, Role: FOLLOWER
I20260927 16:42:39.131767 18031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4867ec7e4040481399d36b1bf41b934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4867ec7e4040481399d36b1bf41b934a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35213 } }
I20260927 16:42:39.131901 18031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4867ec7e4040481399d36b1bf41b934a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:42:39.131959 18031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4867ec7e4040481399d36b1bf41b934a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:42:39.132076 18031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4867ec7e4040481399d36b1bf41b934a [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:39.132848 18031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4867ec7e4040481399d36b1bf41b934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4867ec7e4040481399d36b1bf41b934a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35213 } }
I20260927 16:42:39.133167 18031 leader_election.cc:302] T 00000000000000000000000000000000 P 4867ec7e4040481399d36b1bf41b934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4867ec7e4040481399d36b1bf41b934a; no voters:
I20260927 16:42:39.133435 18031 leader_election.cc:288] T 00000000000000000000000000000000 P 4867ec7e4040481399d36b1bf41b934a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:42:39.133540 18036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4867ec7e4040481399d36b1bf41b934a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:42:39.133765 18036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4867ec7e4040481399d36b1bf41b934a [term 1 LEADER]: Becoming Leader. State: Replica: 4867ec7e4040481399d36b1bf41b934a, State: Running, Role: LEADER
I20260927 16:42:39.134059 18036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4867ec7e4040481399d36b1bf41b934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4867ec7e4040481399d36b1bf41b934a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35213 } }
I20260927 16:42:39.134569 18031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4867ec7e4040481399d36b1bf41b934a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:42:39.135537 18037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4867ec7e4040481399d36b1bf41b934a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4867ec7e4040481399d36b1bf41b934a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4867ec7e4040481399d36b1bf41b934a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35213 } } }
I20260927 16:42:39.135685 18037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4867ec7e4040481399d36b1bf41b934a [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:39.135783 18038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4867ec7e4040481399d36b1bf41b934a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4867ec7e4040481399d36b1bf41b934a. Latest consensus state: current_term: 1 leader_uuid: "4867ec7e4040481399d36b1bf41b934a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4867ec7e4040481399d36b1bf41b934a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35213 } } }
I20260927 16:42:39.135880 18038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4867ec7e4040481399d36b1bf41b934a [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:39.136250 18045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:42:39.138667 18045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:42:39.143541 18045 catalog_manager.cc:1347] Generated new cluster ID: 5c4dc8b2fd2643b09240a8f22ad00dc4
I20260927 16:42:39.143602 18045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:42:39.171422 18045 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:42:39.172562 18045 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:42:39.181638 18045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4867ec7e4040481399d36b1bf41b934a: Generated new TSK 0
I20260927 16:42:39.182402 18045 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:42:39.197813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35213
--builtin_ntp_servers=127.0.98.20:36231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 16:42:39.314462 18055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:39.314682 18055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:39.314716 18055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:39.318570 18055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:39.318616 18055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:39.318703 18055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:42:39.323189 18055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:39.324198 18055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:39.325798 18055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:39.334493 18063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:39.334527 18060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:39.334553 18061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:39.334707 18055 server_base.cc:1034] running on GCE node
I20260927 16:42:39.335266 18055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:39.335978 18055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:39.337132 18055 hybrid_clock.cc:648] HybridClock initialized: now 1790527359337120 us; error 23 us; skew 500 ppm
I20260927 16:42:39.338742 18055 webserver.cc:460] Webserver started at http://127.0.98.1:36441/ using document root <none> and password file <none>
I20260927 16:42:39.339334 18055 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:39.339407 18055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:39.339622 18055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:39.341423 18055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-0/data/instance:
uuid: "2f5409bbcac841fc9db7a51a4458521f"
format_stamp: "Formatted at 2026-09-27 16:42:39 on dist-test-slave-z40b"
server_key: "93ca44f673180bdfa1946d1b29cc1feb"
server_key_iv: "a7c3edfae51cecf680a402fa8b1b7efc"
server_key_version: "encryptionkey@0"
I20260927 16:42:39.341943 18055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2f5409bbcac841fc9db7a51a4458521f"
format_stamp: "Formatted at 2026-09-27 16:42:39 on dist-test-slave-z40b"
server_key: "93ca44f673180bdfa1946d1b29cc1feb"
server_key_iv: "a7c3edfae51cecf680a402fa8b1b7efc"
server_key_version: "encryptionkey@0"
I20260927 16:42:39.345257 18055 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:42:39.347510 18069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:39.348634 18055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:42:39.348762 18055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "2f5409bbcac841fc9db7a51a4458521f"
format_stamp: "Formatted at 2026-09-27 16:42:39 on dist-test-slave-z40b"
server_key: "93ca44f673180bdfa1946d1b29cc1feb"
server_key_iv: "a7c3edfae51cecf680a402fa8b1b7efc"
server_key_version: "encryptionkey@0"
I20260927 16:42:39.348855 18055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:39.379336 18055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:39.379992 18055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:39.380182 18055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:39.380744 18055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:42:39.381634 18055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:42:39.381678 18055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:39.381744 18055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:42:39.381773 18055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:39.390125 18055 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44967
I20260927 16:42:39.390152 18182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44967 every 8 connection(s)
I20260927 16:42:39.390976 18055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:42:39.394579 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 18055
I20260927 16:42:39.394722 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:42:39.395015 392 external_mini_cluster.cc:1428] Setting key b9e06edc593221f58bbe473103e635c1
I20260927 16:42:39.396850 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35213
--builtin_ntp_servers=127.0.98.20:36231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 16:42:39.400207 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:39.392664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60857 (local address 127.0.98.62:35213)
0927 16:42:39.392808 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:39.392812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:39.393867 (+ 1055us) server_negotiation.cc:407] Connection header received
0927 16:42:39.395345 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:39.395349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:39.395403 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:39.395494 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:39.396339 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.396855 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.398247 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.398490 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.398601 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:39.399762 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:39.399765 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:39.399768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:42:39.399779 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:39.399798 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:42:39.399806 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:39.399878 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:39.400038 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 16:42:39.400099 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 16:42:39.401379 18185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:39.393093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35213 (local address 127.0.98.1:60857)
0927 16:42:39.393703 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:39.393753 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:39.395071 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:39.395668 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:39.395675 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:39.395755 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:42:39.396205 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:39.396217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.396991 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.396998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:39.398130 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:39.398139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.398636 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.398640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:39.398812 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:39.399467 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:42:39.399472 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:42:39.399525 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:42:39.399601 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:42:39.399888 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:42:39.399900 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:42:39.399908 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 16:42:39.400033 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 16:42:39.400319 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":113,"threads_started":1}
I20260927 16:42:39.402666 18183 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35213
I20260927 16:42:39.402901 18183 heartbeater.cc:464] Registering TS with master...
I20260927 16:42:39.403434 18183 heartbeater.cc:511] Master 127.0.98.62:35213 requested a full tablet report, sending...
I20260927 16:42:39.404860 17995 ts_manager.cc:194] Registered new tserver with Master: 2f5409bbcac841fc9db7a51a4458521f (127.0.98.1:44967)
I20260927 16:42:39.405944 17995 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:60857
W20260927 16:42:39.503976 18186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:39.504263 18186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:39.504335 18186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:39.508354 18186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:39.508430 18186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:39.508522 18186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:42:39.513168 18186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:39.514194 18186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:39.515625 18186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:39.525144 18192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:39.525156 18194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:39.525156 18191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:39.525429 18186 server_base.cc:1034] running on GCE node
I20260927 16:42:39.525887 18186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:39.526525 18186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:39.527704 18186 hybrid_clock.cc:648] HybridClock initialized: now 1790527359527663 us; error 53 us; skew 500 ppm
I20260927 16:42:39.529361 18186 webserver.cc:460] Webserver started at http://127.0.98.2:45625/ using document root <none> and password file <none>
I20260927 16:42:39.529907 18186 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:39.530011 18186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:39.530248 18186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:39.532274 18186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-1/data/instance:
uuid: "43b9cd89f5e1442dbe20e62941deb029"
format_stamp: "Formatted at 2026-09-27 16:42:39 on dist-test-slave-z40b"
server_key: "241a0dafd94bb6a6c0c747e8719057ef"
server_key_iv: "1536eacbf0db17134f2cf7dd03b274c1"
server_key_version: "encryptionkey@0"
I20260927 16:42:39.532809 18186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-1/wal/instance:
uuid: "43b9cd89f5e1442dbe20e62941deb029"
format_stamp: "Formatted at 2026-09-27 16:42:39 on dist-test-slave-z40b"
server_key: "241a0dafd94bb6a6c0c747e8719057ef"
server_key_iv: "1536eacbf0db17134f2cf7dd03b274c1"
server_key_version: "encryptionkey@0"
I20260927 16:42:39.536486 18186 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:42:39.538712 18200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:39.539899 18186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:42:39.540035 18186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "43b9cd89f5e1442dbe20e62941deb029"
format_stamp: "Formatted at 2026-09-27 16:42:39 on dist-test-slave-z40b"
server_key: "241a0dafd94bb6a6c0c747e8719057ef"
server_key_iv: "1536eacbf0db17134f2cf7dd03b274c1"
server_key_version: "encryptionkey@0"
I20260927 16:42:39.540140 18186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:39.553114 18186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:39.553692 18186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:39.553862 18186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:39.554354 18186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:42:39.555179 18186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:42:39.555318 18186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:39.555387 18186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:42:39.555418 18186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:39.564841 18186 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39133
I20260927 16:42:39.564858 18313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39133 every 8 connection(s)
I20260927 16:42:39.565687 18186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:42:39.572860 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 18186
I20260927 16:42:39.572996 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:42:39.573308 392 external_mini_cluster.cc:1428] Setting key 0e302785f3619c8ceaed6dc25bba7dc5
I20260927 16:42:39.574757 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:39.567578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44947 (local address 127.0.98.62:35213)
0927 16:42:39.567734 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:39.567738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:39.568727 (+ 989us) server_negotiation.cc:407] Connection header received
0927 16:42:39.570017 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:39.570019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:39.570064 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:39.570138 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:39.571005 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.571576 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.572675 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.572882 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.573018 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:39.574247 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:39.574250 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:39.574252 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:42:39.574262 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:39.574280 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:42:39.574288 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:39.574340 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:39.574511 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 16:42:39.574590 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 16:42:39.575512 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35213
--builtin_ntp_servers=127.0.98.20:36231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 16:42:39.575927 18316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:39.567979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35213 (local address 127.0.98.2:44947)
0927 16:42:39.568588 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:39.568627 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:39.569837 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:39.570322 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:39.570328 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:39.570410 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:42:39.570883 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:39.570895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.571682 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.571685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:39.572547 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:39.572555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.573066 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.573070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:39.573271 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:39.573962 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:42:39.573963 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:42:39.574004 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:42:39.574100 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:42:39.574367 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:42:39.574375 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:42:39.574381 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 16:42:39.574519 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 16:42:39.574822 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":104,"threads_started":1}
I20260927 16:42:39.577015 18314 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35213
I20260927 16:42:39.577280 18314 heartbeater.cc:464] Registering TS with master...
I20260927 16:42:39.577781 18314 heartbeater.cc:511] Master 127.0.98.62:35213 requested a full tablet report, sending...
I20260927 16:42:39.578841 17995 ts_manager.cc:194] Registered new tserver with Master: 43b9cd89f5e1442dbe20e62941deb029 (127.0.98.2:39133)
I20260927 16:42:39.579700 17995 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:44947
W20260927 16:42:39.684232 18317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:39.684456 18317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:39.684490 18317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:39.688374 18317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:39.688421 18317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:39.688489 18317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:42:39.693017 18317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:39.693919 18317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:39.695592 18317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:39.704732 18322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:39.704734 18325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:39.704735 18323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:39.705066 18317 server_base.cc:1034] running on GCE node
I20260927 16:42:39.705461 18317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:39.706120 18317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:39.707288 18317 hybrid_clock.cc:648] HybridClock initialized: now 1790527359707250 us; error 46 us; skew 500 ppm
I20260927 16:42:39.709141 18317 webserver.cc:460] Webserver started at http://127.0.98.3:38477/ using document root <none> and password file <none>
I20260927 16:42:39.709715 18317 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:39.709806 18317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:39.710036 18317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:39.712184 18317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-2/data/instance:
uuid: "5bc150589fcc40b48ac81a108407fb2e"
format_stamp: "Formatted at 2026-09-27 16:42:39 on dist-test-slave-z40b"
server_key: "986a4ee6099368efd5cacd76fed01eb2"
server_key_iv: "76d3ad8a42b7cc5525ac8925b3f34bc6"
server_key_version: "encryptionkey@0"
I20260927 16:42:39.712765 18317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5bc150589fcc40b48ac81a108407fb2e"
format_stamp: "Formatted at 2026-09-27 16:42:39 on dist-test-slave-z40b"
server_key: "986a4ee6099368efd5cacd76fed01eb2"
server_key_iv: "76d3ad8a42b7cc5525ac8925b3f34bc6"
server_key_version: "encryptionkey@0"
I20260927 16:42:39.716444 18317 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:42:39.718897 18331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:39.720396 18317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:42:39.720554 18317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "5bc150589fcc40b48ac81a108407fb2e"
format_stamp: "Formatted at 2026-09-27 16:42:39 on dist-test-slave-z40b"
server_key: "986a4ee6099368efd5cacd76fed01eb2"
server_key_iv: "76d3ad8a42b7cc5525ac8925b3f34bc6"
server_key_version: "encryptionkey@0"
I20260927 16:42:39.720676 18317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:39.747330 18317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:39.748006 18317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:39.748191 18317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:39.748764 18317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:42:39.749656 18317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:42:39.749732 18317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:39.749800 18317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:42:39.749848 18317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:39.757962 18317 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33583
I20260927 16:42:39.758023 18444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33583 every 8 connection(s)
I20260927 16:42:39.758817 18317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:42:39.762663 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 18317
I20260927 16:42:39.762746 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:42:39.763021 392 external_mini_cluster.cc:1428] Setting key b24064cc23b942c5ffe0e75cd4fa3498
I20260927 16:42:39.767596 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:39.760575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34885 (local address 127.0.98.62:35213)
0927 16:42:39.760703 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:39.760707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:39.761683 (+ 976us) server_negotiation.cc:407] Connection header received
0927 16:42:39.763028 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:39.763034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:39.763097 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:39.763191 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:39.764252 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.764812 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.765929 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.766161 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.766237 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:39.767173 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:39.767176 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:39.767178 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:42:39.767188 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:39.767253 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:42:39.767268 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:39.767324 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:39.767448 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 16:42:39.767492 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:42:39.768409 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:39.760949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35213 (local address 127.0.98.3:34885)
0927 16:42:39.761545 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:39.761583 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:39.762831 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:39.763416 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:39.763425 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:39.763543 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:42:39.764120 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:39.764140 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.764937 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.764940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:39.765827 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:39.765835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.766283 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.766285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:39.766420 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:39.766952 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:42:39.766953 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:42:39.766991 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:42:39.767048 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:42:39.767327 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:42:39.767332 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:42:39.767335 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 16:42:39.767425 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 16:42:39.767654 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":102,"threads_started":1}
I20260927 16:42:39.769415 18445 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35213
I20260927 16:42:39.769624 18445 heartbeater.cc:464] Registering TS with master...
I20260927 16:42:39.770110 18445 heartbeater.cc:511] Master 127.0.98.62:35213 requested a full tablet report, sending...
I20260927 16:42:39.771034 17995 ts_manager.cc:194] Registered new tserver with Master: 5bc150589fcc40b48ac81a108407fb2e (127.0.98.3:33583)
I20260927 16:42:39.771620 17995 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:34885
I20260927 16:42:39.776564 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:42:39.781502 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:39.778367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49680 (local address 127.0.98.62:35213)
0927 16:42:39.778473 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:39.778477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:39.778646 (+ 169us) server_negotiation.cc:407] Connection header received
0927 16:42:39.778730 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:39.778732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:39.778763 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:39.778817 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:39.779039 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.779560 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.780312 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.780503 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.780572 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:39.781214 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:39.781216 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:39.781218 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:42:39.781228 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:39.781243 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:42:39.781250 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:39.781300 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:39.781354 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 16:42:39.781392 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:42:39.783911 17995 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:42:39.786679 17995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:42:39.800362 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:39.794875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33583 (local address 127.0.0.1:37182)
0927 16:42:39.795914 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:39.795927 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:39.796027 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:39.796454 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:39.796460 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:39.796480 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:39.796570 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:39.796575 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.797564 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.797567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:39.799037 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:39.799047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.799872 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.799876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:39.799974 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:39.800008 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 16:42:39.800110 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 16:42:39.800218 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":980,"mutex_wait_us":38,"thread_start_us":77,"threads_started":1}
I20260927 16:42:39.800370 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:39.795333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37182 (local address 127.0.98.3:33583)
0927 16:42:39.795679 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:39.795691 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:39.796006 (+ 315us) server_negotiation.cc:407] Connection header received
0927 16:42:39.796139 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:39.796143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:39.796306 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:39.796452 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:39.796672 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.797444 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.799167 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.799739 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.799874 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:39.800060 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:39.800136 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 16:42:39.800219 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":81,"threads_started":1}
I20260927 16:42:39.800782 18457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:39.794639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44967 (local address 127.0.0.1:60256)
0927 16:42:39.795818 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:39.795831 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:39.795950 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:39.796611 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:39.796614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:39.796637 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:39.796733 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:39.796739 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.798075 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.798079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:39.799088 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:39.799120 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.800338 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.800342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:39.800487 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:39.800521 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 16:42:39.800597 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 16:42:39.800654 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1109,"mutex_wait_us":70,"thread_start_us":96,"threads_started":1}
I20260927 16:42:39.800875 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:39.795267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60256 (local address 127.0.98.1:44967)
0927 16:42:39.795679 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:39.795691 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:39.795926 (+ 235us) server_negotiation.cc:407] Connection header received
0927 16:42:39.796130 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:39.796135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:39.796306 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:39.796452 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:39.796860 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.797935 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.799237 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.800019 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.800118 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:39.800275 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:39.800636 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 16:42:39.800735 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":130,"threads_started":1}
I20260927 16:42:39.801294 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:39.794472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39133 (local address 127.0.0.1:45592)
0927 16:42:39.795318 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:39.795368 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:39.795547 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:39.796269 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:39.796274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:39.796292 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:39.796375 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:39.796378 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.797766 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.797769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:39.799052 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:39.799057 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.800338 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.800342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:39.800796 (+ 454us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:39.801015 (+ 219us) client_negotiation.cc:769] Sending connection context
0927 16:42:39.801097 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 16:42:39.801175 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":380,"thread_start_us":78,"threads_started":1}
I20260927 16:42:39.801661 18458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:39.794629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45592 (local address 127.0.98.2:39133)
0927 16:42:39.795072 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:39.795083 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:39.795448 (+ 365us) server_negotiation.cc:407] Connection header received
0927 16:42:39.795727 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:39.795734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:39.795952 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:39.796096 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:39.796517 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.797657 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.799198 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:39.800014 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:39.800118 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:39.800275 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:39.801222 (+ 947us) server_negotiation.cc:299] Negotiation successful
0927 16:42:39.801319 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":133,"threads_started":1}
I20260927 16:42:39.803429 18379 tablet_service.cc:1467] Processing CreateTablet for tablet b9b9c347659d49fa9e4d54d22903bfd3 (DEFAULT_TABLE table=test-table [id=493eca911713423aa6a0baa5b2141bb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:42:39.804324 18379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9b9c347659d49fa9e4d54d22903bfd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:39.804091 18117 tablet_service.cc:1467] Processing CreateTablet for tablet b9b9c347659d49fa9e4d54d22903bfd3 (DEFAULT_TABLE table=test-table [id=493eca911713423aa6a0baa5b2141bb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:42:39.804091 18248 tablet_service.cc:1467] Processing CreateTablet for tablet b9b9c347659d49fa9e4d54d22903bfd3 (DEFAULT_TABLE table=test-table [id=493eca911713423aa6a0baa5b2141bb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:42:39.805068 18248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9b9c347659d49fa9e4d54d22903bfd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:39.805069 18117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9b9c347659d49fa9e4d54d22903bfd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:39.810372 18462 tablet_bootstrap.cc:492] T b9b9c347659d49fa9e4d54d22903bfd3 P 5bc150589fcc40b48ac81a108407fb2e: Bootstrap starting.
I20260927 16:42:39.811700 18464 tablet_bootstrap.cc:492] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029: Bootstrap starting.
I20260927 16:42:39.813069 18462 tablet_bootstrap.cc:654] T b9b9c347659d49fa9e4d54d22903bfd3 P 5bc150589fcc40b48ac81a108407fb2e: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:39.813454 18463 tablet_bootstrap.cc:492] T b9b9c347659d49fa9e4d54d22903bfd3 P 2f5409bbcac841fc9db7a51a4458521f: Bootstrap starting.
I20260927 16:42:39.813745 18464 tablet_bootstrap.cc:654] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:39.813902 18462 log.cc:824] T b9b9c347659d49fa9e4d54d22903bfd3 P 5bc150589fcc40b48ac81a108407fb2e: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:39.814514 18464 log.cc:824] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:39.815546 18463 tablet_bootstrap.cc:654] T b9b9c347659d49fa9e4d54d22903bfd3 P 2f5409bbcac841fc9db7a51a4458521f: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:39.816212 18463 log.cc:824] T b9b9c347659d49fa9e4d54d22903bfd3 P 2f5409bbcac841fc9db7a51a4458521f: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:39.816241 18462 tablet_bootstrap.cc:492] T b9b9c347659d49fa9e4d54d22903bfd3 P 5bc150589fcc40b48ac81a108407fb2e: No bootstrap required, opened a new log
I20260927 16:42:39.816435 18462 ts_tablet_manager.cc:1397] T b9b9c347659d49fa9e4d54d22903bfd3 P 5bc150589fcc40b48ac81a108407fb2e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260927 16:42:39.816574 18464 tablet_bootstrap.cc:492] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029: No bootstrap required, opened a new log
I20260927 16:42:39.816771 18464 ts_tablet_manager.cc:1397] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:42:39.817766 18463 tablet_bootstrap.cc:492] T b9b9c347659d49fa9e4d54d22903bfd3 P 2f5409bbcac841fc9db7a51a4458521f: No bootstrap required, opened a new log
I20260927 16:42:39.817953 18463 ts_tablet_manager.cc:1397] T b9b9c347659d49fa9e4d54d22903bfd3 P 2f5409bbcac841fc9db7a51a4458521f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 16:42:39.820730 18464 raft_consensus.cc:348] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b9cd89f5e1442dbe20e62941deb029" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39133 } } peers { permanent_uuid: "2f5409bbcac841fc9db7a51a4458521f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44967 } } peers { permanent_uuid: "5bc150589fcc40b48ac81a108407fb2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33583 } }
I20260927 16:42:39.820730 18462 raft_consensus.cc:348] T b9b9c347659d49fa9e4d54d22903bfd3 P 5bc150589fcc40b48ac81a108407fb2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b9cd89f5e1442dbe20e62941deb029" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39133 } } peers { permanent_uuid: "2f5409bbcac841fc9db7a51a4458521f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44967 } } peers { permanent_uuid: "5bc150589fcc40b48ac81a108407fb2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33583 } }
I20260927 16:42:39.820987 18462 raft_consensus.cc:374] T b9b9c347659d49fa9e4d54d22903bfd3 P 5bc150589fcc40b48ac81a108407fb2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:39.820987 18464 raft_consensus.cc:374] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:39.821039 18464 raft_consensus.cc:729] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43b9cd89f5e1442dbe20e62941deb029, State: Initialized, Role: FOLLOWER
I20260927 16:42:39.821048 18462 raft_consensus.cc:729] T b9b9c347659d49fa9e4d54d22903bfd3 P 5bc150589fcc40b48ac81a108407fb2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bc150589fcc40b48ac81a108407fb2e, State: Initialized, Role: FOLLOWER
I20260927 16:42:39.820883 18463 raft_consensus.cc:348] T b9b9c347659d49fa9e4d54d22903bfd3 P 2f5409bbcac841fc9db7a51a4458521f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b9cd89f5e1442dbe20e62941deb029" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39133 } } peers { permanent_uuid: "2f5409bbcac841fc9db7a51a4458521f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44967 } } peers { permanent_uuid: "5bc150589fcc40b48ac81a108407fb2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33583 } }
I20260927 16:42:39.821189 18463 raft_consensus.cc:374] T b9b9c347659d49fa9e4d54d22903bfd3 P 2f5409bbcac841fc9db7a51a4458521f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:39.821271 18463 raft_consensus.cc:729] T b9b9c347659d49fa9e4d54d22903bfd3 P 2f5409bbcac841fc9db7a51a4458521f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f5409bbcac841fc9db7a51a4458521f, State: Initialized, Role: FOLLOWER
I20260927 16:42:39.821558 18464 consensus_queue.cc:260] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b9cd89f5e1442dbe20e62941deb029" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39133 } } peers { permanent_uuid: "2f5409bbcac841fc9db7a51a4458521f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44967 } } peers { permanent_uuid: "5bc150589fcc40b48ac81a108407fb2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33583 } }
I20260927 16:42:39.821558 18462 consensus_queue.cc:260] T b9b9c347659d49fa9e4d54d22903bfd3 P 5bc150589fcc40b48ac81a108407fb2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b9cd89f5e1442dbe20e62941deb029" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39133 } } peers { permanent_uuid: "2f5409bbcac841fc9db7a51a4458521f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44967 } } peers { permanent_uuid: "5bc150589fcc40b48ac81a108407fb2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33583 } }
I20260927 16:42:39.821771 18463 consensus_queue.cc:260] T b9b9c347659d49fa9e4d54d22903bfd3 P 2f5409bbcac841fc9db7a51a4458521f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b9cd89f5e1442dbe20e62941deb029" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39133 } } peers { permanent_uuid: "2f5409bbcac841fc9db7a51a4458521f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44967 } } peers { permanent_uuid: "5bc150589fcc40b48ac81a108407fb2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33583 } }
I20260927 16:42:39.822525 18464 ts_tablet_manager.cc:1428] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 16:42:39.822525 18462 ts_tablet_manager.cc:1428] T b9b9c347659d49fa9e4d54d22903bfd3 P 5bc150589fcc40b48ac81a108407fb2e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 16:42:39.822770 18463 ts_tablet_manager.cc:1428] T b9b9c347659d49fa9e4d54d22903bfd3 P 2f5409bbcac841fc9db7a51a4458521f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 16:42:39.822916 18445 heartbeater.cc:503] Master 127.0.98.62:35213 was elected leader, sending a full tablet report...
I20260927 16:42:39.823112 18183 heartbeater.cc:503] Master 127.0.98.62:35213 was elected leader, sending a full tablet report...
I20260927 16:42:39.823364 18314 heartbeater.cc:503] Master 127.0.98.62:35213 was elected leader, sending a full tablet report...
W20260927 16:42:39.892176 18184 tablet.cc:2369] T b9b9c347659d49fa9e4d54d22903bfd3 P 2f5409bbcac841fc9db7a51a4458521f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:42:40.009912 18446 tablet.cc:2369] T b9b9c347659d49fa9e4d54d22903bfd3 P 5bc150589fcc40b48ac81a108407fb2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:42:40.066823 18315 tablet.cc:2369] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:42:40.140614 18469 raft_consensus.cc:482] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:42:40.140841 18469 raft_consensus.cc:504] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b9cd89f5e1442dbe20e62941deb029" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39133 } } peers { permanent_uuid: "2f5409bbcac841fc9db7a51a4458521f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44967 } } peers { permanent_uuid: "5bc150589fcc40b48ac81a108407fb2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33583 } }
I20260927 16:42:40.141986 18469 leader_election.cc:288] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f5409bbcac841fc9db7a51a4458521f (127.0.98.1:44967), 5bc150589fcc40b48ac81a108407fb2e (127.0.98.3:33583)
I20260927 16:42:40.145865 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:40.142287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43593 (local address 127.0.98.1:44967)
0927 16:42:40.142416 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:40.142422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:40.142625 (+ 203us) server_negotiation.cc:407] Connection header received
0927 16:42:40.142778 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:40.142781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:40.142838 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:40.142916 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:40.143257 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:40.143787 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:40.144953 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:40.145320 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:40.145437 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:40.145496 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:40.145669 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 16:42:40.145748 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:42:40.145884 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:40.142211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44967 (local address 127.0.98.2:43593)
0927 16:42:40.142514 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:40.142529 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:40.142656 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:40.142939 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:40.142942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:40.142957 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:40.143126 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:40.143133 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:40.143887 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:40.143894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:40.144832 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:40.144841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:40.145445 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:40.145449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:40.145549 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:40.145567 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 16:42:40.145654 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 16:42:40.145739 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":23,"thread_start_us":78,"threads_started":1}
I20260927 16:42:40.146052 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:40.142335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52377 (local address 127.0.98.3:33583)
0927 16:42:40.142502 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:40.142506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:40.142669 (+ 163us) server_negotiation.cc:407] Connection header received
0927 16:42:40.142825 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:40.142828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:40.142875 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:40.142942 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:40.143364 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:40.143950 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:40.145077 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:40.145478 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:40.145598 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:40.145652 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:40.145840 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 16:42:40.145906 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 16:42:40.146196 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:40.142229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33583 (local address 127.0.98.2:52377)
0927 16:42:40.142555 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:40.142566 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:40.142694 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:40.143047 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:40.143050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:40.143064 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:40.143241 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:40.143248 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:40.144083 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:40.144088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:40.144958 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:40.144967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:40.145609 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:40.145612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:40.145725 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:40.145741 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 16:42:40.145825 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 16:42:40.145885 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":23,"thread_start_us":66,"threads_started":1}
I20260927 16:42:40.146860 18399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b9c347659d49fa9e4d54d22903bfd3" candidate_uuid: "43b9cd89f5e1442dbe20e62941deb029" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bc150589fcc40b48ac81a108407fb2e" is_pre_election: true
I20260927 16:42:40.146839 18137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b9c347659d49fa9e4d54d22903bfd3" candidate_uuid: "43b9cd89f5e1442dbe20e62941deb029" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f5409bbcac841fc9db7a51a4458521f" is_pre_election: true
I20260927 16:42:40.147161 18137 raft_consensus.cc:2454] T b9b9c347659d49fa9e4d54d22903bfd3 P 2f5409bbcac841fc9db7a51a4458521f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43b9cd89f5e1442dbe20e62941deb029 in term 0.
I20260927 16:42:40.147161 18399 raft_consensus.cc:2454] T b9b9c347659d49fa9e4d54d22903bfd3 P 5bc150589fcc40b48ac81a108407fb2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43b9cd89f5e1442dbe20e62941deb029 in term 0.
I20260927 16:42:40.147706 18202 leader_election.cc:302] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43b9cd89f5e1442dbe20e62941deb029, 5bc150589fcc40b48ac81a108407fb2e; no voters:
I20260927 16:42:40.147971 18469 raft_consensus.cc:2781] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:42:40.148051 18469 raft_consensus.cc:482] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:42:40.148093 18469 raft_consensus.cc:3037] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:40.149091 18469 raft_consensus.cc:504] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b9cd89f5e1442dbe20e62941deb029" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39133 } } peers { permanent_uuid: "2f5409bbcac841fc9db7a51a4458521f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44967 } } peers { permanent_uuid: "5bc150589fcc40b48ac81a108407fb2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33583 } }
I20260927 16:42:40.149421 18469 leader_election.cc:288] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029 [CANDIDATE]: Term 1 election: Requested vote from peers 2f5409bbcac841fc9db7a51a4458521f (127.0.98.1:44967), 5bc150589fcc40b48ac81a108407fb2e (127.0.98.3:33583)
I20260927 16:42:40.149718 18137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b9c347659d49fa9e4d54d22903bfd3" candidate_uuid: "43b9cd89f5e1442dbe20e62941deb029" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f5409bbcac841fc9db7a51a4458521f"
I20260927 16:42:40.149744 18399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b9c347659d49fa9e4d54d22903bfd3" candidate_uuid: "43b9cd89f5e1442dbe20e62941deb029" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bc150589fcc40b48ac81a108407fb2e"
I20260927 16:42:40.149823 18137 raft_consensus.cc:3037] T b9b9c347659d49fa9e4d54d22903bfd3 P 2f5409bbcac841fc9db7a51a4458521f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:40.149879 18399 raft_consensus.cc:3037] T b9b9c347659d49fa9e4d54d22903bfd3 P 5bc150589fcc40b48ac81a108407fb2e [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:40.150784 18137 raft_consensus.cc:2454] T b9b9c347659d49fa9e4d54d22903bfd3 P 2f5409bbcac841fc9db7a51a4458521f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43b9cd89f5e1442dbe20e62941deb029 in term 1.
I20260927 16:42:40.150925 18399 raft_consensus.cc:2454] T b9b9c347659d49fa9e4d54d22903bfd3 P 5bc150589fcc40b48ac81a108407fb2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43b9cd89f5e1442dbe20e62941deb029 in term 1.
I20260927 16:42:40.151142 18203 leader_election.cc:302] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f5409bbcac841fc9db7a51a4458521f, 43b9cd89f5e1442dbe20e62941deb029; no voters:
I20260927 16:42:40.151335 18469 raft_consensus.cc:2781] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:42:40.151542 18469 raft_consensus.cc:686] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029 [term 1 LEADER]: Becoming Leader. State: Replica: 43b9cd89f5e1442dbe20e62941deb029, State: Running, Role: LEADER
I20260927 16:42:40.151876 18469 consensus_queue.cc:237] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b9cd89f5e1442dbe20e62941deb029" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39133 } } peers { permanent_uuid: "2f5409bbcac841fc9db7a51a4458521f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44967 } } peers { permanent_uuid: "5bc150589fcc40b48ac81a108407fb2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33583 } }
I20260927 16:42:40.154987 17993 catalog_manager.cc:5495] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43b9cd89f5e1442dbe20e62941deb029 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "43b9cd89f5e1442dbe20e62941deb029" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b9cd89f5e1442dbe20e62941deb029" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f5409bbcac841fc9db7a51a4458521f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bc150589fcc40b48ac81a108407fb2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33583 } health_report { overall_health: UNKNOWN } } }
I20260927 16:42:40.247635 18458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:40.244513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45604 (local address 127.0.98.2:39133)
0927 16:42:40.244654 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:40.244659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:40.244671 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:42:40.244762 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:40.244764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:40.244820 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:40.244907 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:42:40.245112 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:40.245582 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:40.246373 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:40.246582 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:40.246657 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:40.246918 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:42:40.247341 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:42:40.247429 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:40.247491 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 16:42:40.247531 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 16:42:40.253361 18137 raft_consensus.cc:1261] T b9b9c347659d49fa9e4d54d22903bfd3 P 2f5409bbcac841fc9db7a51a4458521f [term 1 FOLLOWER]: Refusing update from remote peer 43b9cd89f5e1442dbe20e62941deb029: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:42:40.253361 18399 raft_consensus.cc:1261] T b9b9c347659d49fa9e4d54d22903bfd3 P 5bc150589fcc40b48ac81a108407fb2e [term 1 FOLLOWER]: Refusing update from remote peer 43b9cd89f5e1442dbe20e62941deb029: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:42:40.254055 18473 consensus_queue.cc:1035] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bc150589fcc40b48ac81a108407fb2e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:42:40.254243 18469 consensus_queue.cc:1035] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f5409bbcac841fc9db7a51a4458521f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:42:40.262506 18477 mvcc.cc:205] Tried to move back new op lower bound from 7334000067590549504 to 7334000067184603136. Current Snapshot: MvccSnapshot[applied={T|T < 7334000067590549504}]
I20260927 16:42:40.270670 17994 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:42:40.270859 17994 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:42:40.273192 17994 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4867ec7e4040481399d36b1bf41b934a: Sending DeleteTablet for 3 replicas of tablet b9b9c347659d49fa9e4d54d22903bfd3
I20260927 16:42:40.273905 18117 tablet_service.cc:1514] Processing DeleteTablet for tablet b9b9c347659d49fa9e4d54d22903bfd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:42:40 UTC) from {username='slave'} at 127.0.0.1:60256
I20260927 16:42:40.274024 18379 tablet_service.cc:1514] Processing DeleteTablet for tablet b9b9c347659d49fa9e4d54d22903bfd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:42:40 UTC) from {username='slave'} at 127.0.0.1:37182
I20260927 16:42:40.274384 18248 tablet_service.cc:1514] Processing DeleteTablet for tablet b9b9c347659d49fa9e4d54d22903bfd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:42:40 UTC) from {username='slave'} at 127.0.0.1:45592
I20260927 16:42:40.274535 18487 tablet_replica.cc:330] T b9b9c347659d49fa9e4d54d22903bfd3 P 5bc150589fcc40b48ac81a108407fb2e: stopping tablet replica
I20260927 16:42:40.274638 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 18055
I20260927 16:42:40.275640 18488 tablet_replica.cc:330] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029: stopping tablet replica
I20260927 16:42:40.275805 18487 raft_consensus.cc:2229] T b9b9c347659d49fa9e4d54d22903bfd3 P 5bc150589fcc40b48ac81a108407fb2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:42:40.275875 18488 raft_consensus.cc:2229] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:42:40.276224 18488 raft_consensus.cc:2258] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:42:40.276216 18487 raft_consensus.cc:2258] T b9b9c347659d49fa9e4d54d22903bfd3 P 5bc150589fcc40b48ac81a108407fb2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:42:40.277614 18487 ts_tablet_manager.cc:1905] T b9b9c347659d49fa9e4d54d22903bfd3 P 5bc150589fcc40b48ac81a108407fb2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:42:40.278184 18488 ts_tablet_manager.cc:1905] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:42:40.280499 18487 ts_tablet_manager.cc:1918] T b9b9c347659d49fa9e4d54d22903bfd3 P 5bc150589fcc40b48ac81a108407fb2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:42:40.280661 18487 log.cc:1196] T b9b9c347659d49fa9e4d54d22903bfd3 P 5bc150589fcc40b48ac81a108407fb2e: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-2/wal/wals/b9b9c347659d49fa9e4d54d22903bfd3
I20260927 16:42:40.280857 18488 ts_tablet_manager.cc:1918] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:42:40.280949 18488 log.cc:1196] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790527121515808-392-0/minicluster-data/ts-1/wal/wals/b9b9c347659d49fa9e4d54d22903bfd3
I20260927 16:42:40.281253 18487 ts_tablet_manager.cc:1939] T b9b9c347659d49fa9e4d54d22903bfd3 P 5bc150589fcc40b48ac81a108407fb2e: Deleting consensus metadata
I20260927 16:42:40.281253 18488 ts_tablet_manager.cc:1939] T b9b9c347659d49fa9e4d54d22903bfd3 P 43b9cd89f5e1442dbe20e62941deb029: Deleting consensus metadata
I20260927 16:42:40.282442 17979 catalog_manager.cc:4841] TS 43b9cd89f5e1442dbe20e62941deb029 (127.0.98.2:39133): tablet b9b9c347659d49fa9e4d54d22903bfd3 (table test-table [id=493eca911713423aa6a0baa5b2141bb3]) successfully deleted
I20260927 16:42:40.282461 17980 catalog_manager.cc:4841] TS 5bc150589fcc40b48ac81a108407fb2e (127.0.98.3:33583): tablet b9b9c347659d49fa9e4d54d22903bfd3 (table test-table [id=493eca911713423aa6a0baa5b2141bb3]) successfully deleted
W20260927 16:42:40.282653 17981 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44967 (error 108)
I20260927 16:42:40.282874 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 18186
I20260927 16:42:40.284816 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:40.284558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44967 (local address 127.0.0.1:60268)
0927 16:42:40.284680 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:40.284749 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 16:42:40.284955 17981 catalog_manager.cc:4584] TS 2f5409bbcac841fc9db7a51a4458521f (127.0.98.1:44967): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b9b9c347659d49fa9e4d54d22903bfd3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44967: connect: Connection refused (error 111)
I20260927 16:42:40.288398 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 18317
I20260927 16:42:40.293656 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 17964
2026-09-27T16:42:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1391 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:42:40 dist-test-slave-z40b krb5kdc[18492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:42:40 dist-test-slave-z40b krb5kdc[18492](info): set up 2 sockets
Sep 27 16:42:40 dist-test-slave-z40b krb5kdc[18492](info): commencing operation
krb5kdc: starting...
W20260927 16:42:41.306949 392 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:42:41 dist-test-slave-z40b krb5kdc[18492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:42:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:42:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:42:41.420171 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46733
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:42:41.529074 18505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:41.529331 18505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:41.529413 18505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:41.533231 18505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:42:41.533304 18505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:41.533342 18505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:41.533396 18505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:42:41.533447 18505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:42:41.538478 18505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:41.539436 18505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:41.540388 18505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:41.548384 18511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:41.548401 18510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:41.548432 18513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:41.548569 18505 server_base.cc:1034] running on GCE node
I20260927 16:42:41.549136 18505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:41.550158 18505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:41.551287 18505 hybrid_clock.cc:648] HybridClock initialized: now 1790527361551248 us; error 45 us; skew 500 ppm
Sep 27 16:42:41 dist-test-slave-z40b krb5kdc[18492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:41.553755 18505 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:42:41.554582 18505 webserver.cc:460] Webserver started at http://127.0.98.62:44239/ using document root <none> and password file <none>
I20260927 16:42:41.555115 18505 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:41.555227 18505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:41.555456 18505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:41.557380 18505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "4be92efa025449cb81d809d54c7692af"
format_stamp: "Formatted at 2026-09-27 16:42:41 on dist-test-slave-z40b"
server_key: "be87a225bb3f2ec37f01bcc2ce45a684"
server_key_iv: "e4e1a91d4e49a025344037ff209a8f31"
server_key_version: "encryptionkey@0"
I20260927 16:42:41.557924 18505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "4be92efa025449cb81d809d54c7692af"
format_stamp: "Formatted at 2026-09-27 16:42:41 on dist-test-slave-z40b"
server_key: "be87a225bb3f2ec37f01bcc2ce45a684"
server_key_iv: "e4e1a91d4e49a025344037ff209a8f31"
server_key_version: "encryptionkey@0"
I20260927 16:42:41.561399 18505 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260927 16:42:41.563551 18520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:41.564634 18505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:42:41.564761 18505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "4be92efa025449cb81d809d54c7692af"
format_stamp: "Formatted at 2026-09-27 16:42:41 on dist-test-slave-z40b"
server_key: "be87a225bb3f2ec37f01bcc2ce45a684"
server_key_iv: "e4e1a91d4e49a025344037ff209a8f31"
server_key_version: "encryptionkey@0"
I20260927 16:42:41.564863 18505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:41.582782 18505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:41.585363 18505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:41.585578 18505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:41.592358 18505 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46733
I20260927 16:42:41.592386 18572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46733 every 8 connection(s)
I20260927 16:42:41.593220 18505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:42:41.595649 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 18505
I20260927 16:42:41.595777 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:42:41.595902 18573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:41.596132 392 external_mini_cluster.cc:1428] Setting key 94ad880f911504e9552b96e8e46f8cae
I20260927 16:42:41.601986 18573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4be92efa025449cb81d809d54c7692af: Bootstrap starting.
I20260927 16:42:41.604169 18573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4be92efa025449cb81d809d54c7692af: Neither blocks nor log segments found. Creating new log.
Sep 27 16:42:41 dist-test-slave-z40b krb5kdc[18492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:42:41.605023 18573 log.cc:824] T 00000000000000000000000000000000 P 4be92efa025449cb81d809d54c7692af: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:41.606887 18573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4be92efa025449cb81d809d54c7692af: No bootstrap required, opened a new log
I20260927 16:42:41.609548 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:41.597469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49478 (local address 127.0.98.62:46733)
0927 16:42:41.597891 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:41.597904 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:41.597942 (+ 38us) server_negotiation.cc:407] Connection header received
0927 16:42:41.598549 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:41.598565 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:41.598863 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:41.599189 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:41.600010 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:41.600969 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:41.601817 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:41.602361 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:41.602526 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:41.605179 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:41.605184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:41.605196 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:41.605221 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:41.606899 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:41.607447 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:41.607452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:41.607456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:41.607557 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:41.607849 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:41.607851 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:41.607852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:41.608211 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:41.608350 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:41.608624 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 16:42:41.608892 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":139,"threads_started":1}
I20260927 16:42:41.610793 18573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4be92efa025449cb81d809d54c7692af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be92efa025449cb81d809d54c7692af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46733 } }
I20260927 16:42:41.611049 18573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4be92efa025449cb81d809d54c7692af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:41.611096 18573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4be92efa025449cb81d809d54c7692af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4be92efa025449cb81d809d54c7692af, State: Initialized, Role: FOLLOWER
I20260927 16:42:41.611545 18573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4be92efa025449cb81d809d54c7692af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be92efa025449cb81d809d54c7692af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46733 } }
I20260927 16:42:41.611718 18573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4be92efa025449cb81d809d54c7692af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:42:41.611790 18573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4be92efa025449cb81d809d54c7692af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:42:41.611912 18573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4be92efa025449cb81d809d54c7692af [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:41.612753 18573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4be92efa025449cb81d809d54c7692af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be92efa025449cb81d809d54c7692af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46733 } }
I20260927 16:42:41.613075 18573 leader_election.cc:302] T 00000000000000000000000000000000 P 4be92efa025449cb81d809d54c7692af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4be92efa025449cb81d809d54c7692af; no voters:
I20260927 16:42:41.613371 18573 leader_election.cc:288] T 00000000000000000000000000000000 P 4be92efa025449cb81d809d54c7692af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:42:41.613519 18578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4be92efa025449cb81d809d54c7692af [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:42:41.613725 18578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4be92efa025449cb81d809d54c7692af [term 1 LEADER]: Becoming Leader. State: Replica: 4be92efa025449cb81d809d54c7692af, State: Running, Role: LEADER
I20260927 16:42:41.614074 18578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4be92efa025449cb81d809d54c7692af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be92efa025449cb81d809d54c7692af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46733 } }
I20260927 16:42:41.614511 18573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4be92efa025449cb81d809d54c7692af [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:42:41.615705 18579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4be92efa025449cb81d809d54c7692af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4be92efa025449cb81d809d54c7692af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be92efa025449cb81d809d54c7692af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46733 } } }
I20260927 16:42:41.615844 18579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4be92efa025449cb81d809d54c7692af [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:41.616161 18580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4be92efa025449cb81d809d54c7692af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4be92efa025449cb81d809d54c7692af. Latest consensus state: current_term: 1 leader_uuid: "4be92efa025449cb81d809d54c7692af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be92efa025449cb81d809d54c7692af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46733 } } }
I20260927 16:42:41.616266 18580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4be92efa025449cb81d809d54c7692af [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:41.616837 18587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:42:41.619874 18587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:42:41.625177 18587 catalog_manager.cc:1347] Generated new cluster ID: 30da6f10196f45bdb184e6bba31fe4f7
I20260927 16:42:41.625236 18587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:42:41.668390 18587 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:42:41.669505 18587 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:42:41.674760 18587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4be92efa025449cb81d809d54c7692af: Generated new TSK 0
I20260927 16:42:41.675527 18587 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:42:41.712672 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46733
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:42:41.816161 18599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:41.816428 18599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:41.816500 18599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:41.820279 18599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:41.820348 18599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:41.820431 18599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:42:41.825110 18599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:41.826006 18599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:41.826969 18599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:41.835652 18605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:41.835809 18599 server_base.cc:1034] running on GCE node
W20260927 16:42:41.835682 18604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:41.835636 18607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:41.836285 18599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:41.836864 18599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:41.838044 18599 hybrid_clock.cc:648] HybridClock initialized: now 1790527361838007 us; error 54 us; skew 500 ppm
Sep 27 16:42:41 dist-test-slave-z40b krb5kdc[18492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:41.847105 18599 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:42:41.847990 18599 webserver.cc:460] Webserver started at http://127.0.98.1:40539/ using document root <none> and password file <none>
I20260927 16:42:41.848522 18599 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:41.848613 18599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:41.848834 18599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:41.850785 18599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-0/data/instance:
uuid: "9d8d47e8615941538056279dfb15019c"
format_stamp: "Formatted at 2026-09-27 16:42:41 on dist-test-slave-z40b"
server_key: "79fd8a01dcdf08e2f17e40735892d5a7"
server_key_iv: "9dbefde26a79ee517bfca8bad5af6cd2"
server_key_version: "encryptionkey@0"
I20260927 16:42:41.851415 18599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9d8d47e8615941538056279dfb15019c"
format_stamp: "Formatted at 2026-09-27 16:42:41 on dist-test-slave-z40b"
server_key: "79fd8a01dcdf08e2f17e40735892d5a7"
server_key_iv: "9dbefde26a79ee517bfca8bad5af6cd2"
server_key_version: "encryptionkey@0"
I20260927 16:42:41.855013 18599 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 16:42:41.857369 18614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:41.858568 18599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:42:41.858696 18599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "9d8d47e8615941538056279dfb15019c"
format_stamp: "Formatted at 2026-09-27 16:42:41 on dist-test-slave-z40b"
server_key: "79fd8a01dcdf08e2f17e40735892d5a7"
server_key_iv: "9dbefde26a79ee517bfca8bad5af6cd2"
server_key_version: "encryptionkey@0"
I20260927 16:42:41.858803 18599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:41.881692 18599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:41.884460 18599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:41.884672 18599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:41.885228 18599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:42:41.885977 18599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:42:41.886044 18599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:41.886108 18599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:42:41.886163 18599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:41.894867 18599 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44329
I20260927 16:42:41.894891 18727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44329 every 8 connection(s)
I20260927 16:42:41.895785 18599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:42:41.899173 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 18599
I20260927 16:42:41.899361 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:42:41.899664 392 external_mini_cluster.cc:1428] Setting key 53d7a02bf6f522c8db546a5972b8ff8d
Sep 27 16:42:41 dist-test-slave-z40b krb5kdc[18492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:42:41.910390 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:41.897495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37771 (local address 127.0.98.62:46733)
0927 16:42:41.897672 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:41.897677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:41.898467 (+ 790us) server_negotiation.cc:407] Connection header received
0927 16:42:41.900117 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:41.900121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:41.900176 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:41.900285 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:41.901551 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:41.902072 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:41.903508 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:41.903932 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:41.904054 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:41.906737 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:41.906744 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:41.906748 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:41.906782 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:41.908495 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:41.908975 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:41.908981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:41.908986 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:41.909072 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:41.909392 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:41.909395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:41.909397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:41.909636 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:41.909740 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:41.910172 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 16:42:41.910266 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 16:42:41.911304 18730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:41.897896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46733 (local address 127.0.98.1:37771)
0927 16:42:41.898320 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:41.898359 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:41.899916 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:41.900472 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:41.900479 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:41.900812 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:42:41.901394 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:41.901413 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:41.902201 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:41.902207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:41.903391 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:41.903400 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:41.904071 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:41.904073 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:41.904204 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:41.904782 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:42:41.904798 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:42:41.906569 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:42:41.908613 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:41.908617 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:41.908629 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:41.908874 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:41.909171 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:41.909173 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:41.909174 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:41.909297 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:41.909738 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:42:41.909742 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:42:41.909963 (+ 221us) client_negotiation.cc:769] Sending connection context
0927 16:42:41.910112 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 16:42:41.910343 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 16:42:41.912602 18728 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46733
I20260927 16:42:41.912853 18728 heartbeater.cc:464] Registering TS with master...
I20260927 16:42:41.913407 18728 heartbeater.cc:511] Master 127.0.98.62:46733 requested a full tablet report, sending...
I20260927 16:42:41.915007 18537 ts_manager.cc:194] Registered new tserver with Master: 9d8d47e8615941538056279dfb15019c (127.0.98.1:44329)
I20260927 16:42:41.916363 18537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:42:41.927618 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46733
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:42:42.033284 18733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:42.033511 18733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:42.033576 18733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:42.037438 18733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:42.037487 18733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:42.037600 18733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:42:42.042745 18733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:42.043753 18733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:42.044760 18733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:42.053737 18741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:42.053733 18738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:42.053717 18739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:42.054117 18733 server_base.cc:1034] running on GCE node
I20260927 16:42:42.054442 18733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:42.055014 18733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:42.056167 18733 hybrid_clock.cc:648] HybridClock initialized: now 1790527362056116 us; error 94 us; skew 500 ppm
Sep 27 16:42:42 dist-test-slave-z40b krb5kdc[18492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:42.058888 18733 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:42:42.059744 18733 webserver.cc:460] Webserver started at http://127.0.98.2:45009/ using document root <none> and password file <none>
I20260927 16:42:42.060278 18733 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:42.060366 18733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:42.060596 18733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:42.062628 18733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-1/data/instance:
uuid: "62768c90ee5148f6a373ccdf9fdf15a8"
format_stamp: "Formatted at 2026-09-27 16:42:42 on dist-test-slave-z40b"
server_key: "f8bfc29ddb80a2e4478c64d361fbfc81"
server_key_iv: "67906018787487045cbd9cbf6d2c6299"
server_key_version: "encryptionkey@0"
I20260927 16:42:42.063184 18733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-1/wal/instance:
uuid: "62768c90ee5148f6a373ccdf9fdf15a8"
format_stamp: "Formatted at 2026-09-27 16:42:42 on dist-test-slave-z40b"
server_key: "f8bfc29ddb80a2e4478c64d361fbfc81"
server_key_iv: "67906018787487045cbd9cbf6d2c6299"
server_key_version: "encryptionkey@0"
I20260927 16:42:42.066677 18733 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:42:42.069064 18748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:42.070305 18733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:42:42.070468 18733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "62768c90ee5148f6a373ccdf9fdf15a8"
format_stamp: "Formatted at 2026-09-27 16:42:42 on dist-test-slave-z40b"
server_key: "f8bfc29ddb80a2e4478c64d361fbfc81"
server_key_iv: "67906018787487045cbd9cbf6d2c6299"
server_key_version: "encryptionkey@0"
I20260927 16:42:42.070585 18733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:42.093371 18733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:42.096639 18733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:42.096836 18733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:42.097404 18733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:42:42.098214 18733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:42:42.098284 18733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:42.098371 18733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:42:42.098415 18733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:42.107038 18733 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34583
I20260927 16:42:42.107064 18861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34583 every 8 connection(s)
I20260927 16:42:42.107928 18733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:42:42.113565 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 18733
I20260927 16:42:42.113686 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:42:42.113965 392 external_mini_cluster.cc:1428] Setting key d295e8b7f1aa88ce6da64ef94bd1d6ab
Sep 27 16:42:42 dist-test-slave-z40b krb5kdc[18492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:42:42.122143 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:42.109651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35761 (local address 127.0.98.62:46733)
0927 16:42:42.109782 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:42.109786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:42.110575 (+ 789us) server_negotiation.cc:407] Connection header received
0927 16:42:42.111891 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:42.111895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:42.111955 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:42.112060 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:42.113172 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.113757 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.115086 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.115446 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.115555 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:42.118137 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:42.118144 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:42.118150 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:42.118183 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:42.119957 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:42.120600 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:42.120603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:42.120604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:42.120677 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:42.121088 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:42.121090 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:42.121091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:42.121247 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:42.121318 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:42.121937 (+ 619us) server_negotiation.cc:299] Negotiation successful
0927 16:42:42.122029 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:42:42.123476 18864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:42.110007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46733 (local address 127.0.98.2:35761)
0927 16:42:42.110430 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:42.110477 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:42.111672 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:42.112232 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:42.112238 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:42.112572 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:42:42.113027 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:42.113040 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.113877 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.113886 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:42.114970 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:42.114978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.115589 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.115593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:42.115750 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:42.116404 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:42:42.116421 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:42:42.117997 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:42:42.120076 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:42.120101 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:42.120119 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:42.120500 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:42.120782 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:42.120787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:42.120791 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:42.120987 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:42.121351 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:42:42.121360 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:42:42.121690 (+ 330us) client_negotiation.cc:769] Sending connection context
0927 16:42:42.121907 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 16:42:42.122362 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 16:42:42.124681 18862 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46733
I20260927 16:42:42.124929 18862 heartbeater.cc:464] Registering TS with master...
I20260927 16:42:42.125453 18862 heartbeater.cc:511] Master 127.0.98.62:46733 requested a full tablet report, sending...
I20260927 16:42:42.126494 18537 ts_manager.cc:194] Registered new tserver with Master: 62768c90ee5148f6a373ccdf9fdf15a8 (127.0.98.2:34583)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 16:42:42.127141 18537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:42:42.140244 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46733
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:42:42.245707 18867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:42.245962 18867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:42.246038 18867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:42.249969 18867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:42.250072 18867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:42.250154 18867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:42:42.255064 18867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:42.256009 18867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:42.256968 18867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:42:42.266292 18867 server_base.cc:1034] running on GCE node
W20260927 16:42:42.266209 18873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:42.266254 18875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:42.266209 18872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:42.266813 18867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:42.267462 18867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:42.268595 18867 hybrid_clock.cc:648] HybridClock initialized: now 1790527362268579 us; error 28 us; skew 500 ppm
Sep 27 16:42:42 dist-test-slave-z40b krb5kdc[18492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:42.271296 18867 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:42:42.272121 18867 webserver.cc:460] Webserver started at http://127.0.98.3:37443/ using document root <none> and password file <none>
I20260927 16:42:42.272650 18867 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:42.272738 18867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:42.272960 18867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:42.275027 18867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-2/data/instance:
uuid: "b494de7bc9ac4b8a8d77a9744d7945a4"
format_stamp: "Formatted at 2026-09-27 16:42:42 on dist-test-slave-z40b"
server_key: "cb93817e541011930e510c601a777b43"
server_key_iv: "4aafc788d3146caba3b83269f3a8f7cb"
server_key_version: "encryptionkey@0"
I20260927 16:42:42.275627 18867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b494de7bc9ac4b8a8d77a9744d7945a4"
format_stamp: "Formatted at 2026-09-27 16:42:42 on dist-test-slave-z40b"
server_key: "cb93817e541011930e510c601a777b43"
server_key_iv: "4aafc788d3146caba3b83269f3a8f7cb"
server_key_version: "encryptionkey@0"
I20260927 16:42:42.279071 18867 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:42:42.281548 18882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:42.282616 18867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:42:42.282742 18867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "b494de7bc9ac4b8a8d77a9744d7945a4"
format_stamp: "Formatted at 2026-09-27 16:42:42 on dist-test-slave-z40b"
server_key: "cb93817e541011930e510c601a777b43"
server_key_iv: "4aafc788d3146caba3b83269f3a8f7cb"
server_key_version: "encryptionkey@0"
I20260927 16:42:42.282842 18867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:42.303980 18867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:42.306743 18867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:42.306902 18867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:42.307466 18867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:42:42.308194 18867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:42:42.308238 18867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:42.308272 18867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:42:42.308287 18867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:42.316876 18867 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38687
I20260927 16:42:42.316893 18995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38687 every 8 connection(s)
I20260927 16:42:42.317715 18867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:42:42.326233 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 18867
I20260927 16:42:42.326393 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:42:42.326685 392 external_mini_cluster.cc:1428] Setting key e1b9ab547e3a3bb9247b264a305d5169
Sep 27 16:42:42 dist-test-slave-z40b krb5kdc[18492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:42:42.330960 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:42.319446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41389 (local address 127.0.98.62:46733)
0927 16:42:42.319601 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:42.319606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:42.320443 (+ 837us) server_negotiation.cc:407] Connection header received
0927 16:42:42.322094 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:42.322097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:42.322143 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:42.322230 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:42.323394 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.323941 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.324980 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.325173 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.325243 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:42.327735 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:42.327739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:42.327741 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:42.327766 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:42.329309 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:42.329745 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:42.329747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:42.329748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:42.329817 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:42.330106 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:42.330108 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:42.330109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:42.330285 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:42.330352 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:42.330762 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 16:42:42.330849 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 16:42:42.331739 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:42.319813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46733 (local address 127.0.98.3:41389)
0927 16:42:42.320273 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:42.320312 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:42.321904 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:42.322376 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:42.322382 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:42.322723 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:42:42.323263 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:42.323286 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.324040 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.324042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:42.324874 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:42.324882 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.325288 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.325291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:42.325419 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:42.325900 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:42:42.325915 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:42:42.327571 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:42:42.329403 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:42.329407 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:42.329417 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:42.329657 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:42.329888 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:42.329890 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:42.329891 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:42.330009 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:42.330362 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:42:42.330366 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:42:42.330569 (+ 203us) client_negotiation.cc:769] Sending connection context
0927 16:42:42.330704 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 16:42:42.330923 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":123,"threads_started":1}
I20260927 16:42:42.332773 18996 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46733
I20260927 16:42:42.332991 18996 heartbeater.cc:464] Registering TS with master...
I20260927 16:42:42.333437 18996 heartbeater.cc:511] Master 127.0.98.62:46733 requested a full tablet report, sending...
I20260927 16:42:42.334352 18537 ts_manager.cc:194] Registered new tserver with Master: b494de7bc9ac4b8a8d77a9744d7945a4 (127.0.98.3:38687)
I20260927 16:42:42.334872 18537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41389
I20260927 16:42:42.340453 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:42:42.349957 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:42.342345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49482 (local address 127.0.98.62:46733)
0927 16:42:42.342519 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:42.342524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:42.342735 (+ 211us) server_negotiation.cc:407] Connection header received
0927 16:42:42.342875 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:42.342878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:42.342917 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:42.343025 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:42.343812 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.344337 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.345192 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.345411 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.345520 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:42.346944 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:42.346948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:42.346950 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:42.346975 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:42.348168 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:42.348678 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:42.348681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:42.348683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:42.348760 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:42.349090 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:42.349093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:42.349095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:42.349290 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:42.349446 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:42.349714 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 16:42:42.349805 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 16:42:42.352779 18537 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:42:42.355779 18537 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:42:42.369831 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:42.364175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38687 (local address 127.0.0.1:57278)
0927 16:42:42.364969 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:42.364983 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:42.365069 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:42.365657 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:42.365660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:42.365673 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:42.365796 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:42.365804 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.366930 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.366936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:42.368365 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:42.368376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.369289 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.369292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:42.369407 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:42.369448 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 16:42:42.369568 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 16:42:42.369684 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":24,"thread_start_us":116,"threads_started":1}
I20260927 16:42:42.369846 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:42.364588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57278 (local address 127.0.98.3:38687)
0927 16:42:42.365283 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:42.365293 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:42.365312 (+ 19us) server_negotiation.cc:407] Connection header received
0927 16:42:42.365372 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:42.365377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:42.365541 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:42.365665 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:42.365920 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.366800 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.368470 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.369187 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.369265 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:42.369400 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:42.369612 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 16:42:42.369698 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":141,"threads_started":1}
I20260927 16:42:42.370317 19007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:42.363724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44329 (local address 127.0.0.1:51940)
0927 16:42:42.364478 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:42.364528 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:42.364653 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:42.366278 (+ 1625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:42.366283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:42.366304 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:42.366400 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:42.366406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.367733 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.367736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:42.368801 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:42.368811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.369912 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.369917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:42.370041 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:42.370061 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 16:42:42.370125 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 16:42:42.370183 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"spinlock_wait_cycles":6144,"thread_start_us":65,"threads_started":1}
I20260927 16:42:42.370850 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:42.363845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51940 (local address 127.0.98.1:44329)
0927 16:42:42.365837 (+ 1992us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:42.365852 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:42.365871 (+ 19us) server_negotiation.cc:407] Connection header received
0927 16:42:42.365949 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:42.365954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:42.366117 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:42.366277 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:42.366518 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.367595 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.368923 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.369645 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.369734 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:42.369850 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:42.370566 (+ 716us) server_negotiation.cc:299] Negotiation successful
0927 16:42:42.370689 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1892,"thread_start_us":104,"threads_started":1}
I20260927 16:42:42.370996 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:42.363873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34583 (local address 127.0.0.1:55174)
0927 16:42:42.364478 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:42.364528 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:42.364653 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:42.365546 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:42.365551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:42.365571 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:42.365749 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:42.365757 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.367195 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.367198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:42.368366 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:42.368390 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.370573 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.370577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:42.370699 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:42.370719 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 16:42:42.370790 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 16:42:42.370849 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":58,"spinlock_wait_cycles":4096,"thread_start_us":108,"threads_started":1}
I20260927 16:42:42.371592 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:42.364331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55174 (local address 127.0.98.2:34583)
0927 16:42:42.365083 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:42.365096 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:42.365120 (+ 24us) server_negotiation.cc:407] Connection header received
0927 16:42:42.365198 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:42.365203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:42.365400 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:42.365548 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:42.365879 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.367064 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.369512 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.370287 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.370406 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:42.370546 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:42.371345 (+ 799us) server_negotiation.cc:299] Negotiation successful
0927 16:42:42.371442 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":103,"threads_started":1}
I20260927 16:42:42.373298 18930 tablet_service.cc:1467] Processing CreateTablet for tablet 8822a82ceed84362873034d790e2b2c9 (DEFAULT_TABLE table=test-table [id=0ebf5286507f494b822b73d637fbe8eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:42:42.373646 18796 tablet_service.cc:1467] Processing CreateTablet for tablet 8822a82ceed84362873034d790e2b2c9 (DEFAULT_TABLE table=test-table [id=0ebf5286507f494b822b73d637fbe8eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:42:42.373697 18662 tablet_service.cc:1467] Processing CreateTablet for tablet 8822a82ceed84362873034d790e2b2c9 (DEFAULT_TABLE table=test-table [id=0ebf5286507f494b822b73d637fbe8eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:42:42.374413 18930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8822a82ceed84362873034d790e2b2c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:42.374428 18796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8822a82ceed84362873034d790e2b2c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:42.374585 18662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8822a82ceed84362873034d790e2b2c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:42.380576 19013 tablet_bootstrap.cc:492] T 8822a82ceed84362873034d790e2b2c9 P 62768c90ee5148f6a373ccdf9fdf15a8: Bootstrap starting.
I20260927 16:42:42.381639 19014 tablet_bootstrap.cc:492] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4: Bootstrap starting.
I20260927 16:42:42.381915 19015 tablet_bootstrap.cc:492] T 8822a82ceed84362873034d790e2b2c9 P 9d8d47e8615941538056279dfb15019c: Bootstrap starting.
I20260927 16:42:42.382601 19013 tablet_bootstrap.cc:654] T 8822a82ceed84362873034d790e2b2c9 P 62768c90ee5148f6a373ccdf9fdf15a8: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:42.383639 19013 log.cc:824] T 8822a82ceed84362873034d790e2b2c9 P 62768c90ee5148f6a373ccdf9fdf15a8: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:42.383756 19014 tablet_bootstrap.cc:654] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:42.383756 19015 tablet_bootstrap.cc:654] T 8822a82ceed84362873034d790e2b2c9 P 9d8d47e8615941538056279dfb15019c: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:42.384930 19014 log.cc:824] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:42.384920 19015 log.cc:824] T 8822a82ceed84362873034d790e2b2c9 P 9d8d47e8615941538056279dfb15019c: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:42.386873 19013 tablet_bootstrap.cc:492] T 8822a82ceed84362873034d790e2b2c9 P 62768c90ee5148f6a373ccdf9fdf15a8: No bootstrap required, opened a new log
I20260927 16:42:42.386873 19014 tablet_bootstrap.cc:492] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4: No bootstrap required, opened a new log
I20260927 16:42:42.386943 19015 tablet_bootstrap.cc:492] T 8822a82ceed84362873034d790e2b2c9 P 9d8d47e8615941538056279dfb15019c: No bootstrap required, opened a new log
I20260927 16:42:42.387056 19013 ts_tablet_manager.cc:1397] T 8822a82ceed84362873034d790e2b2c9 P 62768c90ee5148f6a373ccdf9fdf15a8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 16:42:42.387056 19014 ts_tablet_manager.cc:1397] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:42:42.387166 19015 ts_tablet_manager.cc:1397] T 8822a82ceed84362873034d790e2b2c9 P 9d8d47e8615941538056279dfb15019c: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 16:42:42.390002 19015 raft_consensus.cc:348] T 8822a82ceed84362873034d790e2b2c9 P 9d8d47e8615941538056279dfb15019c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d8d47e8615941538056279dfb15019c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44329 } } peers { permanent_uuid: "62768c90ee5148f6a373ccdf9fdf15a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34583 } } peers { permanent_uuid: "b494de7bc9ac4b8a8d77a9744d7945a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38687 } }
I20260927 16:42:42.390359 19015 raft_consensus.cc:374] T 8822a82ceed84362873034d790e2b2c9 P 9d8d47e8615941538056279dfb15019c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:42.390430 19015 raft_consensus.cc:729] T 8822a82ceed84362873034d790e2b2c9 P 9d8d47e8615941538056279dfb15019c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d8d47e8615941538056279dfb15019c, State: Initialized, Role: FOLLOWER
I20260927 16:42:42.390877 19015 consensus_queue.cc:260] T 8822a82ceed84362873034d790e2b2c9 P 9d8d47e8615941538056279dfb15019c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d8d47e8615941538056279dfb15019c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44329 } } peers { permanent_uuid: "62768c90ee5148f6a373ccdf9fdf15a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34583 } } peers { permanent_uuid: "b494de7bc9ac4b8a8d77a9744d7945a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38687 } }
I20260927 16:42:42.390838 19014 raft_consensus.cc:348] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d8d47e8615941538056279dfb15019c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44329 } } peers { permanent_uuid: "62768c90ee5148f6a373ccdf9fdf15a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34583 } } peers { permanent_uuid: "b494de7bc9ac4b8a8d77a9744d7945a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38687 } }
I20260927 16:42:42.390838 19013 raft_consensus.cc:348] T 8822a82ceed84362873034d790e2b2c9 P 62768c90ee5148f6a373ccdf9fdf15a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d8d47e8615941538056279dfb15019c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44329 } } peers { permanent_uuid: "62768c90ee5148f6a373ccdf9fdf15a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34583 } } peers { permanent_uuid: "b494de7bc9ac4b8a8d77a9744d7945a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38687 } }
I20260927 16:42:42.391084 19014 raft_consensus.cc:374] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:42.391084 19013 raft_consensus.cc:374] T 8822a82ceed84362873034d790e2b2c9 P 62768c90ee5148f6a373ccdf9fdf15a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:42.391142 19013 raft_consensus.cc:729] T 8822a82ceed84362873034d790e2b2c9 P 62768c90ee5148f6a373ccdf9fdf15a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62768c90ee5148f6a373ccdf9fdf15a8, State: Initialized, Role: FOLLOWER
I20260927 16:42:42.391149 19014 raft_consensus.cc:729] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b494de7bc9ac4b8a8d77a9744d7945a4, State: Initialized, Role: FOLLOWER
I20260927 16:42:42.391570 18728 heartbeater.cc:503] Master 127.0.98.62:46733 was elected leader, sending a full tablet report...
I20260927 16:42:42.391685 19014 consensus_queue.cc:260] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d8d47e8615941538056279dfb15019c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44329 } } peers { permanent_uuid: "62768c90ee5148f6a373ccdf9fdf15a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34583 } } peers { permanent_uuid: "b494de7bc9ac4b8a8d77a9744d7945a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38687 } }
I20260927 16:42:42.391865 19015 ts_tablet_manager.cc:1428] T 8822a82ceed84362873034d790e2b2c9 P 9d8d47e8615941538056279dfb15019c: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260927 16:42:42.392585 18996 heartbeater.cc:503] Master 127.0.98.62:46733 was elected leader, sending a full tablet report...
I20260927 16:42:42.392820 19013 consensus_queue.cc:260] T 8822a82ceed84362873034d790e2b2c9 P 62768c90ee5148f6a373ccdf9fdf15a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d8d47e8615941538056279dfb15019c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44329 } } peers { permanent_uuid: "62768c90ee5148f6a373ccdf9fdf15a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34583 } } peers { permanent_uuid: "b494de7bc9ac4b8a8d77a9744d7945a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38687 } }
I20260927 16:42:42.393461 18862 heartbeater.cc:503] Master 127.0.98.62:46733 was elected leader, sending a full tablet report...
I20260927 16:42:42.392580 19014 ts_tablet_manager.cc:1428] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260927 16:42:42.394616 19013 ts_tablet_manager.cc:1428] T 8822a82ceed84362873034d790e2b2c9 P 62768c90ee5148f6a373ccdf9fdf15a8: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
W20260927 16:42:42.396868 18729 tablet.cc:2369] T 8822a82ceed84362873034d790e2b2c9 P 9d8d47e8615941538056279dfb15019c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:42:42.413825 19020 raft_consensus.cc:482] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:42:42.413990 19020 raft_consensus.cc:504] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d8d47e8615941538056279dfb15019c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44329 } } peers { permanent_uuid: "62768c90ee5148f6a373ccdf9fdf15a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34583 } } peers { permanent_uuid: "b494de7bc9ac4b8a8d77a9744d7945a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38687 } }
I20260927 16:42:42.415189 19020 leader_election.cc:288] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d8d47e8615941538056279dfb15019c (127.0.98.1:44329), 62768c90ee5148f6a373ccdf9fdf15a8 (127.0.98.2:34583)
I20260927 16:42:42.419850 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:42.415583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37367 (local address 127.0.98.1:44329)
0927 16:42:42.415728 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:42.415733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:42.416076 (+ 343us) server_negotiation.cc:407] Connection header received
0927 16:42:42.416260 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:42.416262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:42.416332 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:42.416411 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:42.416629 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.417310 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.418623 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.419245 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.419343 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:42.419407 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:42.419668 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 16:42:42.419732 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 16:42:42.419850 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:42.415697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53903 (local address 127.0.98.2:34583)
0927 16:42:42.415808 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:42.415813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:42.415828 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:42:42.415880 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:42.415886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:42.415936 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:42.416019 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:42.416344 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.417147 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.418623 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.419238 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.419343 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:42.419407 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:42.419668 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 16:42:42.419732 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:42:42.419874 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:42.415391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34583 (local address 127.0.98.3:53903)
0927 16:42:42.415505 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:42.415523 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:42.415599 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:42.416048 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:42.416054 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:42.416075 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:42.416208 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:42.416214 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.417251 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.417253 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:42.418492 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:42.418503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.419354 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.419357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:42.419560 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:42.419575 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 16:42:42.419637 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 16:42:42.419721 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":74}
I20260927 16:42:42.419890 19022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:42.415391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44329 (local address 127.0.98.3:37367)
0927 16:42:42.415990 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:42.416004 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:42.416126 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:42.416425 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:42.416428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:42.416443 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:42.416527 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:42.416531 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.417405 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.417408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:42.418511 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:42.418517 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.419354 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.419357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:42.419507 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:42.419559 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 16:42:42.419637 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 16:42:42.419721 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":323,"thread_start_us":111,"threads_started":1}
I20260927 16:42:42.420516 18816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8822a82ceed84362873034d790e2b2c9" candidate_uuid: "b494de7bc9ac4b8a8d77a9744d7945a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62768c90ee5148f6a373ccdf9fdf15a8" is_pre_election: true
I20260927 16:42:42.420609 18682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8822a82ceed84362873034d790e2b2c9" candidate_uuid: "b494de7bc9ac4b8a8d77a9744d7945a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d8d47e8615941538056279dfb15019c" is_pre_election: true
I20260927 16:42:42.420807 18816 raft_consensus.cc:2454] T 8822a82ceed84362873034d790e2b2c9 P 62768c90ee5148f6a373ccdf9fdf15a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b494de7bc9ac4b8a8d77a9744d7945a4 in term 0.
I20260927 16:42:42.420921 18682 raft_consensus.cc:2454] T 8822a82ceed84362873034d790e2b2c9 P 9d8d47e8615941538056279dfb15019c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b494de7bc9ac4b8a8d77a9744d7945a4 in term 0.
I20260927 16:42:42.421324 18886 leader_election.cc:302] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62768c90ee5148f6a373ccdf9fdf15a8, b494de7bc9ac4b8a8d77a9744d7945a4; no voters:
I20260927 16:42:42.421590 19020 raft_consensus.cc:2781] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:42:42.421671 19020 raft_consensus.cc:482] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:42:42.421700 19020 raft_consensus.cc:3037] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:42.422727 19020 raft_consensus.cc:504] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d8d47e8615941538056279dfb15019c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44329 } } peers { permanent_uuid: "62768c90ee5148f6a373ccdf9fdf15a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34583 } } peers { permanent_uuid: "b494de7bc9ac4b8a8d77a9744d7945a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38687 } }
I20260927 16:42:42.423094 19020 leader_election.cc:288] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4 [CANDIDATE]: Term 1 election: Requested vote from peers 9d8d47e8615941538056279dfb15019c (127.0.98.1:44329), 62768c90ee5148f6a373ccdf9fdf15a8 (127.0.98.2:34583)
I20260927 16:42:42.423403 18682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8822a82ceed84362873034d790e2b2c9" candidate_uuid: "b494de7bc9ac4b8a8d77a9744d7945a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d8d47e8615941538056279dfb15019c"
I20260927 16:42:42.423430 18816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8822a82ceed84362873034d790e2b2c9" candidate_uuid: "b494de7bc9ac4b8a8d77a9744d7945a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62768c90ee5148f6a373ccdf9fdf15a8"
I20260927 16:42:42.423534 18816 raft_consensus.cc:3037] T 8822a82ceed84362873034d790e2b2c9 P 62768c90ee5148f6a373ccdf9fdf15a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:42.423563 18682 raft_consensus.cc:3037] T 8822a82ceed84362873034d790e2b2c9 P 9d8d47e8615941538056279dfb15019c [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:42.424516 18682 raft_consensus.cc:2454] T 8822a82ceed84362873034d790e2b2c9 P 9d8d47e8615941538056279dfb15019c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b494de7bc9ac4b8a8d77a9744d7945a4 in term 1.
I20260927 16:42:42.424573 18816 raft_consensus.cc:2454] T 8822a82ceed84362873034d790e2b2c9 P 62768c90ee5148f6a373ccdf9fdf15a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b494de7bc9ac4b8a8d77a9744d7945a4 in term 1.
I20260927 16:42:42.424834 18883 leader_election.cc:302] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d8d47e8615941538056279dfb15019c, b494de7bc9ac4b8a8d77a9744d7945a4; no voters:
I20260927 16:42:42.424985 19020 raft_consensus.cc:2781] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:42:42.425197 19020 raft_consensus.cc:686] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4 [term 1 LEADER]: Becoming Leader. State: Replica: b494de7bc9ac4b8a8d77a9744d7945a4, State: Running, Role: LEADER
I20260927 16:42:42.425529 19020 consensus_queue.cc:237] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d8d47e8615941538056279dfb15019c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44329 } } peers { permanent_uuid: "62768c90ee5148f6a373ccdf9fdf15a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34583 } } peers { permanent_uuid: "b494de7bc9ac4b8a8d77a9744d7945a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38687 } }
I20260927 16:42:42.428601 18537 catalog_manager.cc:5495] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b494de7bc9ac4b8a8d77a9744d7945a4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b494de7bc9ac4b8a8d77a9744d7945a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d8d47e8615941538056279dfb15019c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62768c90ee5148f6a373ccdf9fdf15a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b494de7bc9ac4b8a8d77a9744d7945a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38687 } health_report { overall_health: HEALTHY } } }
I20260927 16:42:42.451516 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:42.448303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57294 (local address 127.0.98.3:38687)
0927 16:42:42.448415 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:42.448419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:42.448477 (+ 58us) server_negotiation.cc:407] Connection header received
0927 16:42:42.448592 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:42.448594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:42.448640 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:42.448756 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:42:42.448943 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.449420 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.450215 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.450415 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.450491 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:42.450752 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:42:42.451102 (+ 350us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:42:42.451218 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:42.451368 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 16:42:42.451398 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:42:42.456878 18816 raft_consensus.cc:1261] T 8822a82ceed84362873034d790e2b2c9 P 62768c90ee5148f6a373ccdf9fdf15a8 [term 1 FOLLOWER]: Refusing update from remote peer b494de7bc9ac4b8a8d77a9744d7945a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:42:42.456878 18682 raft_consensus.cc:1261] T 8822a82ceed84362873034d790e2b2c9 P 9d8d47e8615941538056279dfb15019c [term 1 FOLLOWER]: Refusing update from remote peer b494de7bc9ac4b8a8d77a9744d7945a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:42:42.457557 19020 consensus_queue.cc:1035] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d8d47e8615941538056279dfb15019c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:42:42.457753 19023 consensus_queue.cc:1035] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62768c90ee5148f6a373ccdf9fdf15a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:42:42.466612 19028 mvcc.cc:205] Tried to move back new op lower bound from 7334000076616413184 to 7334000076497592320. Current Snapshot: MvccSnapshot[applied={T|T < 7334000076616413184}]
I20260927 16:42:42.473379 18535 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:42:42.473543 18535 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:42:42.475993 18535 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4be92efa025449cb81d809d54c7692af: Sending DeleteTablet for 3 replicas of tablet 8822a82ceed84362873034d790e2b2c9
I20260927 16:42:42.476742 18662 tablet_service.cc:1514] Processing DeleteTablet for tablet 8822a82ceed84362873034d790e2b2c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:42:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51940
I20260927 16:42:42.476742 18796 tablet_service.cc:1514] Processing DeleteTablet for tablet 8822a82ceed84362873034d790e2b2c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:42:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55174
I20260927 16:42:42.477018 18930 tablet_service.cc:1514] Processing DeleteTablet for tablet 8822a82ceed84362873034d790e2b2c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:42:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57278
I20260927 16:42:42.477246 19036 tablet_replica.cc:330] T 8822a82ceed84362873034d790e2b2c9 P 9d8d47e8615941538056279dfb15019c: stopping tablet replica
I20260927 16:42:42.477507 19038 tablet_replica.cc:330] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4: stopping tablet replica
I20260927 16:42:42.477612 19036 raft_consensus.cc:2229] T 8822a82ceed84362873034d790e2b2c9 P 9d8d47e8615941538056279dfb15019c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:42:42.477819 19037 tablet_replica.cc:330] T 8822a82ceed84362873034d790e2b2c9 P 62768c90ee5148f6a373ccdf9fdf15a8: stopping tablet replica
I20260927 16:42:42.477895 19036 raft_consensus.cc:2258] T 8822a82ceed84362873034d790e2b2c9 P 9d8d47e8615941538056279dfb15019c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:42:42.478152 19037 raft_consensus.cc:2229] T 8822a82ceed84362873034d790e2b2c9 P 62768c90ee5148f6a373ccdf9fdf15a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:42:42.478405 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 18599
I20260927 16:42:42.478437 19037 raft_consensus.cc:2258] T 8822a82ceed84362873034d790e2b2c9 P 62768c90ee5148f6a373ccdf9fdf15a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:42:42.480834 19038 raft_consensus.cc:2229] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:42:42.481279 19038 raft_consensus.cc:2258] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:42:42.482753 19038 ts_tablet_manager.cc:1905] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 16:42:42.486306 18521 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44329 (error 108)
I20260927 16:42:42.486351 19038 ts_tablet_manager.cc:1918] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:42:42.486429 19038 log.cc:1196] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790527121515808-392-0/minicluster-data/ts-2/wal/wals/8822a82ceed84362873034d790e2b2c9
I20260927 16:42:42.486817 19038 ts_tablet_manager.cc:1939] T 8822a82ceed84362873034d790e2b2c9 P b494de7bc9ac4b8a8d77a9744d7945a4: Deleting consensus metadata
I20260927 16:42:42.487255 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 18733
I20260927 16:42:42.489195 18523 catalog_manager.cc:4841] TS b494de7bc9ac4b8a8d77a9744d7945a4 (127.0.98.3:38687): tablet 8822a82ceed84362873034d790e2b2c9 (table test-table [id=0ebf5286507f494b822b73d637fbe8eb]) successfully deleted
I20260927 16:42:42.491935 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:42.491687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44329 (local address 127.0.0.1:51944)
0927 16:42:42.491804 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:42.491868 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 16:42:42.492143 18521 catalog_manager.cc:4584] TS 9d8d47e8615941538056279dfb15019c (127.0.98.1:44329): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8822a82ceed84362873034d790e2b2c9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44329: connect: Connection refused (error 111)
I20260927 16:42:42.493806 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 18867
I20260927 16:42:42.495316 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:42.494054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34583 (local address 127.0.0.1:55180)
0927 16:42:42.495163 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:42.495241 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1024}
I20260927 16:42:42.499728 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 18505
2026-09-27T16:42:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2206 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T16:42:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:42:42Z Disabled control of system clock
I20260927 16:42:42.528477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46257
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 16:42:42.636469 19043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:42.636700 19043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:42.636734 19043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:42.640465 19043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:42:42.640507 19043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:42.640522 19043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:42.640553 19043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:42:42.640568 19043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:42:42.644807 19043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46257
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:42.645732 19043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:42.647070 19043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:42.654888 19049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:42.654908 19048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:42.654974 19043 server_base.cc:1034] running on GCE node
W20260927 16:42:42.654902 19051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:42.655705 19043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:42.656704 19043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:42.657856 19043 hybrid_clock.cc:648] HybridClock initialized: now 1790527362657831 us; error 37 us; skew 500 ppm
I20260927 16:42:42.659579 19043 webserver.cc:460] Webserver started at http://127.0.98.62:37513/ using document root <none> and password file <none>
I20260927 16:42:42.660076 19043 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:42.660130 19043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:42.660277 19043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:42.662133 19043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "a2a7388ffc4a41dda17832b0b81c9c4c"
format_stamp: "Formatted at 2026-09-27 16:42:42 on dist-test-slave-z40b"
server_key: "e6ae3ee018a6f402638c8ea0fe0c0031"
server_key_iv: "779d21ac6df3c03896cdd65f58c49db1"
server_key_version: "encryptionkey@0"
I20260927 16:42:42.662595 19043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "a2a7388ffc4a41dda17832b0b81c9c4c"
format_stamp: "Formatted at 2026-09-27 16:42:42 on dist-test-slave-z40b"
server_key: "e6ae3ee018a6f402638c8ea0fe0c0031"
server_key_iv: "779d21ac6df3c03896cdd65f58c49db1"
server_key_version: "encryptionkey@0"
I20260927 16:42:42.666041 19043 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:42:42.668218 19057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:42.669358 19043 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:42:42.669448 19043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "a2a7388ffc4a41dda17832b0b81c9c4c"
format_stamp: "Formatted at 2026-09-27 16:42:42 on dist-test-slave-z40b"
server_key: "e6ae3ee018a6f402638c8ea0fe0c0031"
server_key_iv: "779d21ac6df3c03896cdd65f58c49db1"
server_key_version: "encryptionkey@0"
I20260927 16:42:42.669525 19043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:42.681888 19043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:42.682406 19043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:42.682539 19043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:42.689014 19043 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46257
I20260927 16:42:42.689024 19109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46257 every 8 connection(s)
I20260927 16:42:42.689860 19043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:42:42.692621 19110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:42.693492 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 19043
I20260927 16:42:42.693614 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:42:42.693902 392 external_mini_cluster.cc:1428] Setting key cc8414ca328cde2849a6a48ad4262a1b
I20260927 16:42:42.698352 19110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a7388ffc4a41dda17832b0b81c9c4c: Bootstrap starting.
I20260927 16:42:42.700407 19110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2a7388ffc4a41dda17832b0b81c9c4c: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:42.701210 19110 log.cc:824] T 00000000000000000000000000000000 P a2a7388ffc4a41dda17832b0b81c9c4c: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:42.702723 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:42.695324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43396 (local address 127.0.98.62:46257)
0927 16:42:42.695909 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:42.695922 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:42.695954 (+ 32us) server_negotiation.cc:407] Connection header received
0927 16:42:42.696562 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:42.696576 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:42.696926 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:42.697242 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:42.697663 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.698582 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.699485 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.699916 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.700082 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:42.701118 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:42.701124 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:42.701142 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:42:42.701546 (+ 404us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:42.701597 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:42:42.701616 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:42.701733 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:42.701849 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 16:42:42.702082 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":130,"threads_started":1}
I20260927 16:42:42.703174 19110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a7388ffc4a41dda17832b0b81c9c4c: No bootstrap required, opened a new log
I20260927 16:42:42.706851 19110 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2a7388ffc4a41dda17832b0b81c9c4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7388ffc4a41dda17832b0b81c9c4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46257 } }
I20260927 16:42:42.707106 19110 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2a7388ffc4a41dda17832b0b81c9c4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:42.707185 19110 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2a7388ffc4a41dda17832b0b81c9c4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a7388ffc4a41dda17832b0b81c9c4c, State: Initialized, Role: FOLLOWER
I20260927 16:42:42.707644 19110 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2a7388ffc4a41dda17832b0b81c9c4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7388ffc4a41dda17832b0b81c9c4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46257 } }
I20260927 16:42:42.707791 19110 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2a7388ffc4a41dda17832b0b81c9c4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:42:42.707860 19110 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2a7388ffc4a41dda17832b0b81c9c4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:42:42.707985 19110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2a7388ffc4a41dda17832b0b81c9c4c [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:42.708982 19110 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2a7388ffc4a41dda17832b0b81c9c4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7388ffc4a41dda17832b0b81c9c4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46257 } }
I20260927 16:42:42.709254 19110 leader_election.cc:302] T 00000000000000000000000000000000 P a2a7388ffc4a41dda17832b0b81c9c4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2a7388ffc4a41dda17832b0b81c9c4c; no voters:
I20260927 16:42:42.709484 19110 leader_election.cc:288] T 00000000000000000000000000000000 P a2a7388ffc4a41dda17832b0b81c9c4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:42:42.709587 19115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2a7388ffc4a41dda17832b0b81c9c4c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:42:42.709797 19115 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2a7388ffc4a41dda17832b0b81c9c4c [term 1 LEADER]: Becoming Leader. State: Replica: a2a7388ffc4a41dda17832b0b81c9c4c, State: Running, Role: LEADER
I20260927 16:42:42.710091 19115 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2a7388ffc4a41dda17832b0b81c9c4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7388ffc4a41dda17832b0b81c9c4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46257 } }
I20260927 16:42:42.710623 19110 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2a7388ffc4a41dda17832b0b81c9c4c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:42:42.711501 19116 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2a7388ffc4a41dda17832b0b81c9c4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2a7388ffc4a41dda17832b0b81c9c4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7388ffc4a41dda17832b0b81c9c4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46257 } } }
I20260927 16:42:42.711577 19117 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2a7388ffc4a41dda17832b0b81c9c4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2a7388ffc4a41dda17832b0b81c9c4c. Latest consensus state: current_term: 1 leader_uuid: "a2a7388ffc4a41dda17832b0b81c9c4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7388ffc4a41dda17832b0b81c9c4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46257 } } }
I20260927 16:42:42.711640 19116 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2a7388ffc4a41dda17832b0b81c9c4c [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:42.711640 19117 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2a7388ffc4a41dda17832b0b81c9c4c [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:42.711970 19124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:42:42.714823 19124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:42:42.720330 19124 catalog_manager.cc:1347] Generated new cluster ID: 3029ba85ea77448c9369d3aced5e39ad
I20260927 16:42:42.720391 19124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:42:42.736764 19124 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:42:42.737637 19124 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:42:42.741355 19124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2a7388ffc4a41dda17832b0b81c9c4c: Generated new TSK 0
I20260927 16:42:42.741935 19124 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:42:42.746210 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46257
--builtin_ntp_servers=127.0.98.20:36229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 16:42:42.873116 19134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:42.873342 19134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:42.873376 19134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:42.877293 19134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:42.877341 19134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:42.877421 19134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:42:42.882220 19134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:42.883186 19134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:42.884559 19134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:42.893509 19142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:42.893568 19140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:42.893573 19139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:42.893863 19134 server_base.cc:1034] running on GCE node
I20260927 16:42:42.894235 19134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:42.894866 19134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:42.896020 19134 hybrid_clock.cc:648] HybridClock initialized: now 1790527362896008 us; error 26 us; skew 500 ppm
I20260927 16:42:42.897814 19134 webserver.cc:460] Webserver started at http://127.0.98.1:41237/ using document root <none> and password file <none>
I20260927 16:42:42.898363 19134 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:42.898429 19134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:42.898671 19134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:42.900626 19134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-0/data/instance:
uuid: "c26ea1a176d546a2bd0550146707b5fe"
format_stamp: "Formatted at 2026-09-27 16:42:42 on dist-test-slave-z40b"
server_key: "fb21fb1cd48eba6213fa0e869e500353"
server_key_iv: "356c34a14b3e828c2cf0d600429e6c07"
server_key_version: "encryptionkey@0"
I20260927 16:42:42.901173 19134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c26ea1a176d546a2bd0550146707b5fe"
format_stamp: "Formatted at 2026-09-27 16:42:42 on dist-test-slave-z40b"
server_key: "fb21fb1cd48eba6213fa0e869e500353"
server_key_iv: "356c34a14b3e828c2cf0d600429e6c07"
server_key_version: "encryptionkey@0"
I20260927 16:42:42.904753 19134 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:42:42.907015 19148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:42.908169 19134 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:42:42.908305 19134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "c26ea1a176d546a2bd0550146707b5fe"
format_stamp: "Formatted at 2026-09-27 16:42:42 on dist-test-slave-z40b"
server_key: "fb21fb1cd48eba6213fa0e869e500353"
server_key_iv: "356c34a14b3e828c2cf0d600429e6c07"
server_key_version: "encryptionkey@0"
I20260927 16:42:42.908412 19134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:42.925011 19134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:42.925630 19134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:42.925817 19134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:42.926368 19134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:42:42.927302 19134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:42:42.927372 19134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:42.927451 19134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:42:42.927495 19134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:42.936618 19134 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43011
I20260927 16:42:42.936641 19261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43011 every 8 connection(s)
I20260927 16:42:42.937470 19134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:42:42.942770 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 19134
I20260927 16:42:42.942911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:42:42.943202 392 external_mini_cluster.cc:1428] Setting key d10bd136fea4904839d024acb47a2979
I20260927 16:42:42.945024 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46257
--builtin_ntp_servers=127.0.98.20:36229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 16:42:42.946569 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:42.939194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50965 (local address 127.0.98.62:46257)
0927 16:42:42.939349 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:42.939353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:42.940365 (+ 1012us) server_negotiation.cc:407] Connection header received
0927 16:42:42.941680 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:42.941683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:42.941739 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:42.941871 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:42.942736 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.943440 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.944521 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.944751 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.944856 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:42.945942 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:42.945946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:42.945949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:42:42.945961 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:42.945981 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:42:42.945992 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:42.946067 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:42.946305 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 16:42:42.946377 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:42:42.947366 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:42.939615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46257 (local address 127.0.98.1:50965)
0927 16:42:42.940226 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:42.940263 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:42.941486 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:42.942041 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:42.942047 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:42.942126 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:42:42.942601 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:42.942613 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.943560 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.943564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:42.944400 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:42.944408 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:42.944890 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:42.944894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:42.945049 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:42.945707 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:42:42.945708 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:42:42.945745 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:42:42.945801 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:42:42.946107 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:42:42.946120 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:42:42.946128 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 16:42:42.946297 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 16:42:42.946515 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":105,"threads_started":1}
I20260927 16:42:42.948606 19262 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46257
I20260927 16:42:42.948836 19262 heartbeater.cc:464] Registering TS with master...
I20260927 16:42:42.949306 19262 heartbeater.cc:511] Master 127.0.98.62:46257 requested a full tablet report, sending...
I20260927 16:42:42.950891 19074 ts_manager.cc:194] Registered new tserver with Master: c26ea1a176d546a2bd0550146707b5fe (127.0.98.1:43011)
I20260927 16:42:42.952608 19074 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:50965
W20260927 16:42:43.054730 19265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:43.054965 19265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:43.055001 19265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:43.058935 19265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:43.058984 19265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:43.059037 19265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:42:43.063688 19265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:43.064558 19265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:43.065712 19265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:43.074764 19273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:43.074772 19271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:43.074770 19270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:43.075047 19265 server_base.cc:1034] running on GCE node
I20260927 16:42:43.075505 19265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:43.076093 19265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:43.077276 19265 hybrid_clock.cc:648] HybridClock initialized: now 1790527363077257 us; error 28 us; skew 500 ppm
I20260927 16:42:43.079136 19265 webserver.cc:460] Webserver started at http://127.0.98.2:33277/ using document root <none> and password file <none>
I20260927 16:42:43.079686 19265 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:43.079746 19265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:43.079993 19265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:43.081933 19265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-1/data/instance:
uuid: "a79ebe16c0de4f618c7ed3048601db24"
format_stamp: "Formatted at 2026-09-27 16:42:43 on dist-test-slave-z40b"
server_key: "b3d043c596227820e6fac3ebc4174339"
server_key_iv: "8a7d6643959dff4250e792e72c7e9351"
server_key_version: "encryptionkey@0"
I20260927 16:42:43.082520 19265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a79ebe16c0de4f618c7ed3048601db24"
format_stamp: "Formatted at 2026-09-27 16:42:43 on dist-test-slave-z40b"
server_key: "b3d043c596227820e6fac3ebc4174339"
server_key_iv: "8a7d6643959dff4250e792e72c7e9351"
server_key_version: "encryptionkey@0"
I20260927 16:42:43.086122 19265 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:42:43.088511 19279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:43.089649 19265 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:42:43.089774 19265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "a79ebe16c0de4f618c7ed3048601db24"
format_stamp: "Formatted at 2026-09-27 16:42:43 on dist-test-slave-z40b"
server_key: "b3d043c596227820e6fac3ebc4174339"
server_key_iv: "8a7d6643959dff4250e792e72c7e9351"
server_key_version: "encryptionkey@0"
I20260927 16:42:43.089880 19265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:43.112227 19265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:43.112875 19265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:43.113090 19265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:43.113667 19265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:42:43.114562 19265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:42:43.114632 19265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:43.114696 19265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:42:43.114748 19265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:43.123308 19265 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43015
I20260927 16:42:43.123347 19392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43015 every 8 connection(s)
I20260927 16:42:43.124162 19265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:42:43.131522 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 19265
I20260927 16:42:43.131656 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:42:43.131930 392 external_mini_cluster.cc:1428] Setting key 99fa69efbc08520accd0e9c1ee3d6913
I20260927 16:42:43.132891 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:43.125837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56927 (local address 127.0.98.62:46257)
0927 16:42:43.125981 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:43.125985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:43.127024 (+ 1039us) server_negotiation.cc:407] Connection header received
0927 16:42:43.128361 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:43.128364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:43.128408 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:43.128485 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:43.129285 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.129791 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.130828 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.131034 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.131107 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:43.132384 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:43.132387 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:43.132389 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:42:43.132399 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:43.132416 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:42:43.132423 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:43.132482 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:43.132690 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 16:42:43.132753 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:42:43.133894 19395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:43.126267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46257 (local address 127.0.98.2:56927)
0927 16:42:43.126861 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:43.126899 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:43.128173 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:43.128633 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:43.128639 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:43.128715 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:42:43.129168 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:43.129179 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.129886 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.129888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:43.130731 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:43.130739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.131149 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.131152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:43.131345 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:43.131965 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:42:43.131966 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:42:43.132003 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:42:43.132103 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:42:43.132515 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:42:43.132527 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:42:43.132536 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 16:42:43.132683 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 16:42:43.132990 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":106,"threads_started":1}
I20260927 16:42:43.133973 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46257
--builtin_ntp_servers=127.0.98.20:36229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 16:42:43.135061 19393 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46257
I20260927 16:42:43.135324 19393 heartbeater.cc:464] Registering TS with master...
I20260927 16:42:43.135812 19393 heartbeater.cc:511] Master 127.0.98.62:46257 requested a full tablet report, sending...
I20260927 16:42:43.136816 19074 ts_manager.cc:194] Registered new tserver with Master: a79ebe16c0de4f618c7ed3048601db24 (127.0.98.2:43015)
I20260927 16:42:43.137655 19074 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:56927
W20260927 16:42:43.254949 19396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:43.255188 19396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:43.255298 19396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:43.259774 19396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:43.259822 19396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:43.259879 19396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:42:43.264880 19396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:43.265846 19396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:43.267189 19396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:43.276685 19402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:43.276690 19401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:43.276952 19396 server_base.cc:1034] running on GCE node
W20260927 16:42:43.276731 19404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:43.277437 19396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:43.278016 19396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:43.279196 19396 hybrid_clock.cc:648] HybridClock initialized: now 1790527363279181 us; error 26 us; skew 500 ppm
I20260927 16:42:43.280969 19396 webserver.cc:460] Webserver started at http://127.0.98.3:43201/ using document root <none> and password file <none>
I20260927 16:42:43.281489 19396 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:43.281571 19396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:43.281777 19396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:43.283771 19396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-2/data/instance:
uuid: "a00ab7abf389447ba4ece21374a210d2"
format_stamp: "Formatted at 2026-09-27 16:42:43 on dist-test-slave-z40b"
server_key: "651d30557b950057642be7b898596d38"
server_key_iv: "0ca121f2fe81871691c77197900c038a"
server_key_version: "encryptionkey@0"
I20260927 16:42:43.284286 19396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a00ab7abf389447ba4ece21374a210d2"
format_stamp: "Formatted at 2026-09-27 16:42:43 on dist-test-slave-z40b"
server_key: "651d30557b950057642be7b898596d38"
server_key_iv: "0ca121f2fe81871691c77197900c038a"
server_key_version: "encryptionkey@0"
I20260927 16:42:43.287696 19396 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:42:43.289945 19410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:43.291157 19396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:42:43.291306 19396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "a00ab7abf389447ba4ece21374a210d2"
format_stamp: "Formatted at 2026-09-27 16:42:43 on dist-test-slave-z40b"
server_key: "651d30557b950057642be7b898596d38"
server_key_iv: "0ca121f2fe81871691c77197900c038a"
server_key_version: "encryptionkey@0"
I20260927 16:42:43.291411 19396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:43.306876 19396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:43.307512 19396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:43.307690 19396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:43.308194 19396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:42:43.309032 19396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:42:43.309098 19396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:43.309155 19396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:42:43.309194 19396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:43.317692 19396 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33145
I20260927 16:42:43.317756 19523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33145 every 8 connection(s)
I20260927 16:42:43.318570 19396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:42:43.320600 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 19396
I20260927 16:42:43.320691 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:42:43.321017 392 external_mini_cluster.cc:1428] Setting key 4f371a7f51bf2a7d4e01cd92b2734712
I20260927 16:42:43.327240 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:43.320309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35395 (local address 127.0.98.62:46257)
0927 16:42:43.320466 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:43.320471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:43.321522 (+ 1051us) server_negotiation.cc:407] Connection header received
0927 16:42:43.322925 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:43.322928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:43.322979 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:43.323055 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:43.323877 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.324364 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.325424 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.325632 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.325730 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:43.326726 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:43.326730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:43.326733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:42:43.326744 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:43.326765 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:42:43.326781 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:43.326878 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:43.327031 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 16:42:43.327085 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 16:42:43.327957 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:43.320717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46257 (local address 127.0.98.3:35395)
0927 16:42:43.321389 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:43.321428 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:43.322735 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:43.323235 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:43.323246 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:43.323331 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:42:43.323756 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:43.323766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.324481 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.324488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:43.325279 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:43.325288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.325759 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.325762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:43.325901 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:43.326493 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:42:43.326495 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:42:43.326529 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:42:43.326584 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:42:43.326879 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:42:43.326890 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:42:43.326898 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 16:42:43.327030 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 16:42:43.327293 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":100,"threads_started":1}
I20260927 16:42:43.328979 19524 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46257
I20260927 16:42:43.329169 19524 heartbeater.cc:464] Registering TS with master...
I20260927 16:42:43.329586 19524 heartbeater.cc:511] Master 127.0.98.62:46257 requested a full tablet report, sending...
I20260927 16:42:43.330410 19074 ts_manager.cc:194] Registered new tserver with Master: a00ab7abf389447ba4ece21374a210d2 (127.0.98.3:33145)
I20260927 16:42:43.330919 19074 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:35395
I20260927 16:42:43.334529 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:42:43.339742 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:43.336402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43406 (local address 127.0.98.62:46257)
0927 16:42:43.336525 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:43.336528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:43.336677 (+ 149us) server_negotiation.cc:407] Connection header received
0927 16:42:43.336752 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:43.336754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:43.336786 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:43.336849 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:43.337077 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.337540 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.338292 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.338507 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.338579 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:43.339402 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:43.339405 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:43.339407 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:42:43.339416 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:43.339431 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:42:43.339437 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:43.339536 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:43.339630 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 16:42:43.339657 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 16:42:43.342141 19074 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:42:43.345058 19074 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:42:43.358333 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:43.352652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43011 (local address 127.0.0.1:54190)
0927 16:42:43.353119 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:43.353170 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:43.353299 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:43.354246 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:43.354252 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:43.354274 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:43.354416 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:43.354425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.355611 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.355614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:43.356858 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:43.356869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.357819 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.357823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:43.357924 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:43.357961 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 16:42:43.358073 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 16:42:43.358199 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":36,"thread_start_us":70,"threads_started":1}
I20260927 16:42:43.358363 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:43.352809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54190 (local address 127.0.98.1:43011)
0927 16:42:43.353743 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:43.353757 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:43.353786 (+ 29us) server_negotiation.cc:407] Connection header received
0927 16:42:43.353860 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:43.353866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:43.354086 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:43.354259 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:43.354563 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.355501 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.357004 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.357703 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.357803 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:43.357982 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:43.358118 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 16:42:43.358209 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":825,"thread_start_us":120,"threads_started":1}
I20260927 16:42:43.358698 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:43.352829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43015 (local address 127.0.0.1:40724)
0927 16:42:43.353649 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:43.353663 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:43.353744 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:43.354422 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:43.354425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:43.354443 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:43.354530 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:43.354540 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.355985 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.355988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:43.357200 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:43.357209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.358212 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.358215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:43.358292 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:43.358311 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 16:42:43.358357 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 16:42:43.358573 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":100,"threads_started":1}
I20260927 16:42:43.359169 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:43.353299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33145 (local address 127.0.0.1:54818)
0927 16:42:43.354506 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:43.354519 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:43.354613 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:43.355034 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:43.355036 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:43.355053 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:43.355144 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:43.355150 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.356484 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.356487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:43.357514 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:43.357524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.358712 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.358717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:43.358814 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:43.358927 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 16:42:43.358980 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 16:42:43.359041 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1130,"mutex_wait_us":28,"thread_start_us":85,"threads_started":1}
I20260927 16:42:43.359397 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:43.353616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54818 (local address 127.0.98.3:33145)
0927 16:42:43.354010 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:43.354020 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:43.354615 (+ 595us) server_negotiation.cc:407] Connection header received
0927 16:42:43.354727 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:43.354733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:43.354882 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:43.355026 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:43.355283 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.356219 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.357653 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.358409 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.358719 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:43.358964 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:43.359009 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 16:42:43.359086 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":135,"threads_started":1}
I20260927 16:42:43.359856 19539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:43.353340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40724 (local address 127.0.98.2:43015)
0927 16:42:43.354019 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:43.354032 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:43.354054 (+ 22us) server_negotiation.cc:407] Connection header received
0927 16:42:43.354128 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:43.354133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:43.354271 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:43.354432 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:43.354692 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.355868 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.357322 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.358094 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.358945 (+ 851us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:43.359127 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:43.359231 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 16:42:43.359421 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":128,"threads_started":1}
I20260927 16:42:43.361292 19196 tablet_service.cc:1467] Processing CreateTablet for tablet 97f44fcabc5c43f8a36eff13bead6047 (DEFAULT_TABLE table=test-table [id=8b1d502e9ee64cc789ac171235858421]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:42:43.361721 19458 tablet_service.cc:1467] Processing CreateTablet for tablet 97f44fcabc5c43f8a36eff13bead6047 (DEFAULT_TABLE table=test-table [id=8b1d502e9ee64cc789ac171235858421]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:42:43.362372 19196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f44fcabc5c43f8a36eff13bead6047. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:43.362636 19458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f44fcabc5c43f8a36eff13bead6047. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:43.362377 19327 tablet_service.cc:1467] Processing CreateTablet for tablet 97f44fcabc5c43f8a36eff13bead6047 (DEFAULT_TABLE table=test-table [id=8b1d502e9ee64cc789ac171235858421]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:42:43.363292 19327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f44fcabc5c43f8a36eff13bead6047. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:43.368299 19541 tablet_bootstrap.cc:492] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2: Bootstrap starting.
I20260927 16:42:43.369722 19542 tablet_bootstrap.cc:492] T 97f44fcabc5c43f8a36eff13bead6047 P c26ea1a176d546a2bd0550146707b5fe: Bootstrap starting.
I20260927 16:42:43.371034 19541 tablet_bootstrap.cc:654] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:43.371512 19542 tablet_bootstrap.cc:654] T 97f44fcabc5c43f8a36eff13bead6047 P c26ea1a176d546a2bd0550146707b5fe: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:43.371524 19543 tablet_bootstrap.cc:492] T 97f44fcabc5c43f8a36eff13bead6047 P a79ebe16c0de4f618c7ed3048601db24: Bootstrap starting.
I20260927 16:42:43.371853 19541 log.cc:824] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:43.372332 19542 log.cc:824] T 97f44fcabc5c43f8a36eff13bead6047 P c26ea1a176d546a2bd0550146707b5fe: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:43.373522 19543 tablet_bootstrap.cc:654] T 97f44fcabc5c43f8a36eff13bead6047 P a79ebe16c0de4f618c7ed3048601db24: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:43.373900 19541 tablet_bootstrap.cc:492] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2: No bootstrap required, opened a new log
I20260927 16:42:43.374090 19541 ts_tablet_manager.cc:1397] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260927 16:42:43.374258 19542 tablet_bootstrap.cc:492] T 97f44fcabc5c43f8a36eff13bead6047 P c26ea1a176d546a2bd0550146707b5fe: No bootstrap required, opened a new log
I20260927 16:42:43.374343 19543 log.cc:824] T 97f44fcabc5c43f8a36eff13bead6047 P a79ebe16c0de4f618c7ed3048601db24: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:43.374420 19542 ts_tablet_manager.cc:1397] T 97f44fcabc5c43f8a36eff13bead6047 P c26ea1a176d546a2bd0550146707b5fe: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 16:42:43.376156 19543 tablet_bootstrap.cc:492] T 97f44fcabc5c43f8a36eff13bead6047 P a79ebe16c0de4f618c7ed3048601db24: No bootstrap required, opened a new log
I20260927 16:42:43.376323 19543 ts_tablet_manager.cc:1397] T 97f44fcabc5c43f8a36eff13bead6047 P a79ebe16c0de4f618c7ed3048601db24: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:42:43.377671 19541 raft_consensus.cc:348] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c26ea1a176d546a2bd0550146707b5fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43011 } } peers { permanent_uuid: "a79ebe16c0de4f618c7ed3048601db24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43015 } } peers { permanent_uuid: "a00ab7abf389447ba4ece21374a210d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260927 16:42:43.377888 19541 raft_consensus.cc:374] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:43.377946 19541 raft_consensus.cc:729] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a00ab7abf389447ba4ece21374a210d2, State: Initialized, Role: FOLLOWER
I20260927 16:42:43.378368 19541 consensus_queue.cc:260] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c26ea1a176d546a2bd0550146707b5fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43011 } } peers { permanent_uuid: "a79ebe16c0de4f618c7ed3048601db24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43015 } } peers { permanent_uuid: "a00ab7abf389447ba4ece21374a210d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260927 16:42:43.378582 19542 raft_consensus.cc:348] T 97f44fcabc5c43f8a36eff13bead6047 P c26ea1a176d546a2bd0550146707b5fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c26ea1a176d546a2bd0550146707b5fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43011 } } peers { permanent_uuid: "a79ebe16c0de4f618c7ed3048601db24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43015 } } peers { permanent_uuid: "a00ab7abf389447ba4ece21374a210d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260927 16:42:43.378804 19542 raft_consensus.cc:374] T 97f44fcabc5c43f8a36eff13bead6047 P c26ea1a176d546a2bd0550146707b5fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:43.378866 19542 raft_consensus.cc:729] T 97f44fcabc5c43f8a36eff13bead6047 P c26ea1a176d546a2bd0550146707b5fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c26ea1a176d546a2bd0550146707b5fe, State: Initialized, Role: FOLLOWER
I20260927 16:42:43.379091 19524 heartbeater.cc:503] Master 127.0.98.62:46257 was elected leader, sending a full tablet report...
I20260927 16:42:43.379323 19541 ts_tablet_manager.cc:1428] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 16:42:43.379310 19542 consensus_queue.cc:260] T 97f44fcabc5c43f8a36eff13bead6047 P c26ea1a176d546a2bd0550146707b5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c26ea1a176d546a2bd0550146707b5fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43011 } } peers { permanent_uuid: "a79ebe16c0de4f618c7ed3048601db24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43015 } } peers { permanent_uuid: "a00ab7abf389447ba4ece21374a210d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260927 16:42:43.379885 19262 heartbeater.cc:503] Master 127.0.98.62:46257 was elected leader, sending a full tablet report...
I20260927 16:42:43.380309 19543 raft_consensus.cc:348] T 97f44fcabc5c43f8a36eff13bead6047 P a79ebe16c0de4f618c7ed3048601db24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c26ea1a176d546a2bd0550146707b5fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43011 } } peers { permanent_uuid: "a79ebe16c0de4f618c7ed3048601db24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43015 } } peers { permanent_uuid: "a00ab7abf389447ba4ece21374a210d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260927 16:42:43.380498 19542 ts_tablet_manager.cc:1428] T 97f44fcabc5c43f8a36eff13bead6047 P c26ea1a176d546a2bd0550146707b5fe: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 16:42:43.380556 19543 raft_consensus.cc:374] T 97f44fcabc5c43f8a36eff13bead6047 P a79ebe16c0de4f618c7ed3048601db24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:43.380604 19543 raft_consensus.cc:729] T 97f44fcabc5c43f8a36eff13bead6047 P a79ebe16c0de4f618c7ed3048601db24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a79ebe16c0de4f618c7ed3048601db24, State: Initialized, Role: FOLLOWER
I20260927 16:42:43.381024 19543 consensus_queue.cc:260] T 97f44fcabc5c43f8a36eff13bead6047 P a79ebe16c0de4f618c7ed3048601db24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c26ea1a176d546a2bd0550146707b5fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43011 } } peers { permanent_uuid: "a79ebe16c0de4f618c7ed3048601db24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43015 } } peers { permanent_uuid: "a00ab7abf389447ba4ece21374a210d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260927 16:42:43.381801 19543 ts_tablet_manager.cc:1428] T 97f44fcabc5c43f8a36eff13bead6047 P a79ebe16c0de4f618c7ed3048601db24: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 16:42:43.382113 19393 heartbeater.cc:503] Master 127.0.98.62:46257 was elected leader, sending a full tablet report...
I20260927 16:42:43.434916 19547 raft_consensus.cc:482] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:42:43.435153 19547 raft_consensus.cc:504] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c26ea1a176d546a2bd0550146707b5fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43011 } } peers { permanent_uuid: "a79ebe16c0de4f618c7ed3048601db24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43015 } } peers { permanent_uuid: "a00ab7abf389447ba4ece21374a210d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260927 16:42:43.436435 19547 leader_election.cc:288] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c26ea1a176d546a2bd0550146707b5fe (127.0.98.1:43011), a79ebe16c0de4f618c7ed3048601db24 (127.0.98.2:43015)
W20260927 16:42:43.438516 19263 tablet.cc:2369] T 97f44fcabc5c43f8a36eff13bead6047 P c26ea1a176d546a2bd0550146707b5fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:42:43.440761 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:43.436734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37387 (local address 127.0.98.1:43011)
0927 16:42:43.436852 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:43.436857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:43.436871 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:42:43.437012 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:43.437015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:43.437060 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:43.437124 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:43.437431 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.438181 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.439622 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.440206 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.440307 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:43.440400 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:43.440586 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 16:42:43.440649 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 16:42:43.440761 19539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:43.436781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34273 (local address 127.0.98.2:43015)
0927 16:42:43.436905 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:43.436909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:43.437182 (+ 273us) server_negotiation.cc:407] Connection header received
0927 16:42:43.437283 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:43.437285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:43.437326 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:43.437415 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:43.437658 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.438379 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.439738 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.440288 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.440384 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:43.440433 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:43.440586 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 16:42:43.440649 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:42:43.440778 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:43.436605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43011 (local address 127.0.98.3:37387)
0927 16:42:43.436791 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:43.436807 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:43.436896 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:43.437169 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:43.437171 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:43.437183 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:43.437315 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:43.437321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.438283 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.438285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:43.439488 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:43.439498 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.440331 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.440333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:43.440441 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:43.440474 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 16:42:43.440546 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 16:42:43.440633 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":14}
I20260927 16:42:43.440793 19550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:43.436691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43015 (local address 127.0.98.3:34273)
0927 16:42:43.437067 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:43.437077 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:43.437182 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:43.437460 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:43.437462 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:43.437477 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:43.437557 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:43.437560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.438490 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.438493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:43.439632 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:43.439645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.440403 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.440406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:43.440481 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:43.440500 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 16:42:43.440560 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 16:42:43.440633 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":50,"thread_start_us":122,"threads_started":1}
I20260927 16:42:43.441469 19216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f44fcabc5c43f8a36eff13bead6047" candidate_uuid: "a00ab7abf389447ba4ece21374a210d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c26ea1a176d546a2bd0550146707b5fe" is_pre_election: true
I20260927 16:42:43.441499 19347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f44fcabc5c43f8a36eff13bead6047" candidate_uuid: "a00ab7abf389447ba4ece21374a210d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a79ebe16c0de4f618c7ed3048601db24" is_pre_election: true
I20260927 16:42:43.441758 19216 raft_consensus.cc:2454] T 97f44fcabc5c43f8a36eff13bead6047 P c26ea1a176d546a2bd0550146707b5fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a00ab7abf389447ba4ece21374a210d2 in term 0.
I20260927 16:42:43.441758 19347 raft_consensus.cc:2454] T 97f44fcabc5c43f8a36eff13bead6047 P a79ebe16c0de4f618c7ed3048601db24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a00ab7abf389447ba4ece21374a210d2 in term 0.
I20260927 16:42:43.442216 19411 leader_election.cc:302] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a00ab7abf389447ba4ece21374a210d2, a79ebe16c0de4f618c7ed3048601db24, c26ea1a176d546a2bd0550146707b5fe; no voters:
I20260927 16:42:43.442480 19547 raft_consensus.cc:2781] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:42:43.442554 19547 raft_consensus.cc:482] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:42:43.442584 19547 raft_consensus.cc:3037] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:43.443718 19547 raft_consensus.cc:504] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c26ea1a176d546a2bd0550146707b5fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43011 } } peers { permanent_uuid: "a79ebe16c0de4f618c7ed3048601db24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43015 } } peers { permanent_uuid: "a00ab7abf389447ba4ece21374a210d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260927 16:42:43.444046 19547 leader_election.cc:288] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2 [CANDIDATE]: Term 1 election: Requested vote from peers c26ea1a176d546a2bd0550146707b5fe (127.0.98.1:43011), a79ebe16c0de4f618c7ed3048601db24 (127.0.98.2:43015)
I20260927 16:42:43.444389 19347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f44fcabc5c43f8a36eff13bead6047" candidate_uuid: "a00ab7abf389447ba4ece21374a210d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a79ebe16c0de4f618c7ed3048601db24"
I20260927 16:42:43.444515 19347 raft_consensus.cc:3037] T 97f44fcabc5c43f8a36eff13bead6047 P a79ebe16c0de4f618c7ed3048601db24 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:43.445493 19347 raft_consensus.cc:2454] T 97f44fcabc5c43f8a36eff13bead6047 P a79ebe16c0de4f618c7ed3048601db24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a00ab7abf389447ba4ece21374a210d2 in term 1.
I20260927 16:42:43.445739 19216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f44fcabc5c43f8a36eff13bead6047" candidate_uuid: "a00ab7abf389447ba4ece21374a210d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c26ea1a176d546a2bd0550146707b5fe"
I20260927 16:42:43.445791 19411 leader_election.cc:302] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a00ab7abf389447ba4ece21374a210d2, a79ebe16c0de4f618c7ed3048601db24; no voters:
I20260927 16:42:43.445866 19216 raft_consensus.cc:3037] T 97f44fcabc5c43f8a36eff13bead6047 P c26ea1a176d546a2bd0550146707b5fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:43.445991 19547 raft_consensus.cc:2781] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:42:43.446237 19547 raft_consensus.cc:686] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2 [term 1 LEADER]: Becoming Leader. State: Replica: a00ab7abf389447ba4ece21374a210d2, State: Running, Role: LEADER
I20260927 16:42:43.446596 19547 consensus_queue.cc:237] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c26ea1a176d546a2bd0550146707b5fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43011 } } peers { permanent_uuid: "a79ebe16c0de4f618c7ed3048601db24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43015 } } peers { permanent_uuid: "a00ab7abf389447ba4ece21374a210d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260927 16:42:43.446996 19216 raft_consensus.cc:2454] T 97f44fcabc5c43f8a36eff13bead6047 P c26ea1a176d546a2bd0550146707b5fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a00ab7abf389447ba4ece21374a210d2 in term 1.
I20260927 16:42:43.450022 19073 catalog_manager.cc:5495] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a00ab7abf389447ba4ece21374a210d2 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a00ab7abf389447ba4ece21374a210d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c26ea1a176d546a2bd0550146707b5fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a79ebe16c0de4f618c7ed3048601db24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a00ab7abf389447ba4ece21374a210d2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } health_report { overall_health: HEALTHY } } }
I20260927 16:42:43.480677 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:43.477209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54820 (local address 127.0.98.3:33145)
0927 16:42:43.477349 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:43.477356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:43.477373 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:42:43.477463 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:43.477466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:43.477520 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:43.477640 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:42:43.477882 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.478363 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.479198 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:43.479439 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:43.479551 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:43.479882 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:42:43.480256 (+ 374us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:42:43.480407 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:43.480496 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 16:42:43.480529 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 16:42:43.486516 19216 raft_consensus.cc:1261] T 97f44fcabc5c43f8a36eff13bead6047 P c26ea1a176d546a2bd0550146707b5fe [term 1 FOLLOWER]: Refusing update from remote peer a00ab7abf389447ba4ece21374a210d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:42:43.486515 19347 raft_consensus.cc:1261] T 97f44fcabc5c43f8a36eff13bead6047 P a79ebe16c0de4f618c7ed3048601db24 [term 1 FOLLOWER]: Refusing update from remote peer a00ab7abf389447ba4ece21374a210d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:42:43.487270 19551 consensus_queue.cc:1035] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c26ea1a176d546a2bd0550146707b5fe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:42:43.487442 19547 consensus_queue.cc:1035] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a79ebe16c0de4f618c7ed3048601db24" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:42:43.504307 19073 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:42:43.504480 19073 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:42:43.506919 19073 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2a7388ffc4a41dda17832b0b81c9c4c: Sending DeleteTablet for 3 replicas of tablet 97f44fcabc5c43f8a36eff13bead6047
I20260927 16:42:43.507683 19327 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f44fcabc5c43f8a36eff13bead6047 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:42:43 UTC) from {username='slave'} at 127.0.0.1:40724
I20260927 16:42:43.507733 19458 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f44fcabc5c43f8a36eff13bead6047 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:42:43 UTC) from {username='slave'} at 127.0.0.1:54818
I20260927 16:42:43.507787 19196 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f44fcabc5c43f8a36eff13bead6047 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:42:43 UTC) from {username='slave'} at 127.0.0.1:54190
I20260927 16:42:43.508317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 19134
I20260927 16:42:43.508935 19565 tablet_replica.cc:330] T 97f44fcabc5c43f8a36eff13bead6047 P a79ebe16c0de4f618c7ed3048601db24: stopping tablet replica
I20260927 16:42:43.509228 19565 raft_consensus.cc:2229] T 97f44fcabc5c43f8a36eff13bead6047 P a79ebe16c0de4f618c7ed3048601db24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:42:43.509522 19565 raft_consensus.cc:2258] T 97f44fcabc5c43f8a36eff13bead6047 P a79ebe16c0de4f618c7ed3048601db24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:42:43.509722 19567 tablet_replica.cc:330] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2: stopping tablet replica
I20260927 16:42:43.509994 19567 raft_consensus.cc:2229] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:42:43.510331 19567 raft_consensus.cc:2258] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:42:43.511117 19565 ts_tablet_manager.cc:1905] T 97f44fcabc5c43f8a36eff13bead6047 P a79ebe16c0de4f618c7ed3048601db24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:42:43.511816 19567 ts_tablet_manager.cc:1905] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:42:43.513695 19565 ts_tablet_manager.cc:1918] T 97f44fcabc5c43f8a36eff13bead6047 P a79ebe16c0de4f618c7ed3048601db24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:42:43.513809 19565 log.cc:1196] T 97f44fcabc5c43f8a36eff13bead6047 P a79ebe16c0de4f618c7ed3048601db24: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-1/wal/wals/97f44fcabc5c43f8a36eff13bead6047
I20260927 16:42:43.514103 19565 ts_tablet_manager.cc:1939] T 97f44fcabc5c43f8a36eff13bead6047 P a79ebe16c0de4f618c7ed3048601db24: Deleting consensus metadata
I20260927 16:42:43.514492 19567 ts_tablet_manager.cc:1918] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:42:43.514583 19567 log.cc:1196] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790527121515808-392-0/minicluster-data/ts-2/wal/wals/97f44fcabc5c43f8a36eff13bead6047
I20260927 16:42:43.514870 19567 ts_tablet_manager.cc:1939] T 97f44fcabc5c43f8a36eff13bead6047 P a00ab7abf389447ba4ece21374a210d2: Deleting consensus metadata
I20260927 16:42:43.515269 19058 catalog_manager.cc:4841] TS a79ebe16c0de4f618c7ed3048601db24 (127.0.98.2:43015): tablet 97f44fcabc5c43f8a36eff13bead6047 (table test-table [id=8b1d502e9ee64cc789ac171235858421]) successfully deleted
I20260927 16:42:43.515892 19060 catalog_manager.cc:4841] TS a00ab7abf389447ba4ece21374a210d2 (127.0.98.3:33145): tablet 97f44fcabc5c43f8a36eff13bead6047 (table test-table [id=8b1d502e9ee64cc789ac171235858421]) successfully deleted
W20260927 16:42:43.521064 19060 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43011 (error 108)
I20260927 16:42:43.521306 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 19265
I20260927 16:42:43.522348 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:43.522057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43011 (local address 127.0.0.1:54192)
0927 16:42:43.522203 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:43.522270 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 16:42:43.522600 19060 catalog_manager.cc:4584] TS c26ea1a176d546a2bd0550146707b5fe (127.0.98.1:43011): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 97f44fcabc5c43f8a36eff13bead6047: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43011: connect: Connection refused (error 111)
I20260927 16:42:43.526459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 19396
I20260927 16:42:43.530966 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 19043
2026-09-27T16:42:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1030 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:42:43 dist-test-slave-z40b krb5kdc[19571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:42:43 dist-test-slave-z40b krb5kdc[19571](info): set up 2 sockets
Sep 27 16:42:43 dist-test-slave-z40b krb5kdc[19571](info): commencing operation
krb5kdc: starting...
W20260927 16:42:44.551059 392 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:42:44 dist-test-slave-z40b krb5kdc[19571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:42:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:42:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:42:44.668382 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34571
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:42:44.774121 19584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:44.774390 19584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:44.774471 19584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:44.778496 19584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:42:44.778587 19584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:44.778625 19584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:44.778680 19584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:42:44.778718 19584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:42:44.783694 19584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:44.784647 19584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:44.785590 19584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:44.793828 19589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:44.793857 19590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:44.793844 19592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:44.794148 19584 server_base.cc:1034] running on GCE node
I20260927 16:42:44.794687 19584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:44.795876 19584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:44.797034 19584 hybrid_clock.cc:648] HybridClock initialized: now 1790527364797007 us; error 37 us; skew 500 ppm
Sep 27 16:42:44 dist-test-slave-z40b krb5kdc[19571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:44.799769 19584 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:42:44.800576 19584 webserver.cc:460] Webserver started at http://127.0.98.62:40083/ using document root <none> and password file <none>
I20260927 16:42:44.801046 19584 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:44.801101 19584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:44.801249 19584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:44.803275 19584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "42e2e55b05bd44ed848a9843e153f665"
format_stamp: "Formatted at 2026-09-27 16:42:44 on dist-test-slave-z40b"
server_key: "0c5a4c83189fe8fece46ddefdcefb75a"
server_key_iv: "38fe95a21ef0319ad8f90618c42153ac"
server_key_version: "encryptionkey@0"
I20260927 16:42:44.803803 19584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "42e2e55b05bd44ed848a9843e153f665"
format_stamp: "Formatted at 2026-09-27 16:42:44 on dist-test-slave-z40b"
server_key: "0c5a4c83189fe8fece46ddefdcefb75a"
server_key_iv: "38fe95a21ef0319ad8f90618c42153ac"
server_key_version: "encryptionkey@0"
I20260927 16:42:44.807282 19584 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 16:42:44.809953 19599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:44.811127 19584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:42:44.811295 19584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "42e2e55b05bd44ed848a9843e153f665"
format_stamp: "Formatted at 2026-09-27 16:42:44 on dist-test-slave-z40b"
server_key: "0c5a4c83189fe8fece46ddefdcefb75a"
server_key_iv: "38fe95a21ef0319ad8f90618c42153ac"
server_key_version: "encryptionkey@0"
I20260927 16:42:44.811424 19584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:44.839689 19584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:44.842994 19584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:44.843222 19584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:44.850308 19584 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34571
I20260927 16:42:44.850332 19651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34571 every 8 connection(s)
I20260927 16:42:44.851269 19584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:42:44.853943 19652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:44.854254 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 19584
I20260927 16:42:44.854382 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:42:44.854678 392 external_mini_cluster.cc:1428] Setting key 267066a932b5c2d4e46cf7c5f6c59d70
I20260927 16:42:44.860291 19652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42e2e55b05bd44ed848a9843e153f665: Bootstrap starting.
I20260927 16:42:44.862558 19652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42e2e55b05bd44ed848a9843e153f665: Neither blocks nor log segments found. Creating new log.
Sep 27 16:42:44 dist-test-slave-z40b krb5kdc[19571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:42:44.863361 19652 log.cc:824] T 00000000000000000000000000000000 P 42e2e55b05bd44ed848a9843e153f665: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:44.865162 19652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42e2e55b05bd44ed848a9843e153f665: No bootstrap required, opened a new log
I20260927 16:42:44.868175 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:44.856071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49836 (local address 127.0.98.62:34571)
0927 16:42:44.856514 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:44.856529 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:44.856564 (+ 35us) server_negotiation.cc:407] Connection header received
0927 16:42:44.857222 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:44.857238 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:44.857559 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:44.857890 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:44.858681 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:44.859711 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:44.860601 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:44.861063 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:44.861234 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:44.863693 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:44.863698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:44.863712 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:44.863737 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:44.865460 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:44.865883 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:44.865889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:44.865893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:44.865996 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:44.866271 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:44.866276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:44.866277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:44.866660 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:44.866867 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:44.867153 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 16:42:44.867498 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":137,"threads_started":1}
I20260927 16:42:44.868641 19652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42e2e55b05bd44ed848a9843e153f665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e2e55b05bd44ed848a9843e153f665" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34571 } }
I20260927 16:42:44.868876 19652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42e2e55b05bd44ed848a9843e153f665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:44.868953 19652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42e2e55b05bd44ed848a9843e153f665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42e2e55b05bd44ed848a9843e153f665, State: Initialized, Role: FOLLOWER
I20260927 16:42:44.869427 19652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42e2e55b05bd44ed848a9843e153f665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e2e55b05bd44ed848a9843e153f665" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34571 } }
I20260927 16:42:44.869589 19652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42e2e55b05bd44ed848a9843e153f665 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:42:44.869656 19652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42e2e55b05bd44ed848a9843e153f665 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:42:44.869747 19652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42e2e55b05bd44ed848a9843e153f665 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:44.870693 19652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42e2e55b05bd44ed848a9843e153f665 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e2e55b05bd44ed848a9843e153f665" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34571 } }
I20260927 16:42:44.870970 19652 leader_election.cc:302] T 00000000000000000000000000000000 P 42e2e55b05bd44ed848a9843e153f665 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42e2e55b05bd44ed848a9843e153f665; no voters:
I20260927 16:42:44.871236 19652 leader_election.cc:288] T 00000000000000000000000000000000 P 42e2e55b05bd44ed848a9843e153f665 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:42:44.871381 19657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42e2e55b05bd44ed848a9843e153f665 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:42:44.871588 19657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42e2e55b05bd44ed848a9843e153f665 [term 1 LEADER]: Becoming Leader. State: Replica: 42e2e55b05bd44ed848a9843e153f665, State: Running, Role: LEADER
I20260927 16:42:44.872376 19657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42e2e55b05bd44ed848a9843e153f665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e2e55b05bd44ed848a9843e153f665" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34571 } }
I20260927 16:42:44.872792 19652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42e2e55b05bd44ed848a9843e153f665 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:42:44.874276 19659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42e2e55b05bd44ed848a9843e153f665 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42e2e55b05bd44ed848a9843e153f665. Latest consensus state: current_term: 1 leader_uuid: "42e2e55b05bd44ed848a9843e153f665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e2e55b05bd44ed848a9843e153f665" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34571 } } }
I20260927 16:42:44.874666 19659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42e2e55b05bd44ed848a9843e153f665 [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:44.874558 19658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42e2e55b05bd44ed848a9843e153f665 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42e2e55b05bd44ed848a9843e153f665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e2e55b05bd44ed848a9843e153f665" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34571 } } }
I20260927 16:42:44.875241 19658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42e2e55b05bd44ed848a9843e153f665 [sys.catalog]: This master's current role is: LEADER
W20260927 16:42:44.877216 19671 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42e2e55b05bd44ed848a9843e153f665: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:42:44.877317 19671 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:42:44.877444 19672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:42:44.878345 19672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:42:44.883714 19672 catalog_manager.cc:1347] Generated new cluster ID: 58461bbea00747ac89128f0c0830dfc7
I20260927 16:42:44.883775 19672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:42:44.910068 19672 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:42:44.910983 19672 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:42:44.923092 19672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42e2e55b05bd44ed848a9843e153f665: Generated new TSK 0
I20260927 16:42:44.923789 19672 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 16:42:44.958560 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34571
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:42:45.079873 19678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:45.080114 19678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:45.080155 19678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:45.084018 19678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:45.084066 19678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:45.084121 19678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:42:45.089103 19678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:45.089982 19678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:45.090869 19678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:45.100036 19684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:45.100026 19686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:45.100068 19683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:45.100193 19678 server_base.cc:1034] running on GCE node
I20260927 16:42:45.100695 19678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:45.101298 19678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:45.102428 19678 hybrid_clock.cc:648] HybridClock initialized: now 1790527365102403 us; error 41 us; skew 500 ppm
Sep 27 16:42:45 dist-test-slave-z40b krb5kdc[19571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:45.105834 19678 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 16:42:45.107003 19678 webserver.cc:460] Webserver started at http://127.0.98.1:44285/ using document root <none> and password file <none>
I20260927 16:42:45.107636 19678 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:45.107726 19678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:45.107930 19678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:45.110540 19678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-0/data/instance:
uuid: "80d51f50278f43acb6547271ab5b8d7b"
format_stamp: "Formatted at 2026-09-27 16:42:45 on dist-test-slave-z40b"
server_key: "68df668d6eed0d352ec80aa0ef75583d"
server_key_iv: "9b8dbe44727231b30631cffadc5ad8fb"
server_key_version: "encryptionkey@0"
I20260927 16:42:45.111267 19678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-0/wal/instance:
uuid: "80d51f50278f43acb6547271ab5b8d7b"
format_stamp: "Formatted at 2026-09-27 16:42:45 on dist-test-slave-z40b"
server_key: "68df668d6eed0d352ec80aa0ef75583d"
server_key_iv: "9b8dbe44727231b30631cffadc5ad8fb"
server_key_version: "encryptionkey@0"
I20260927 16:42:45.115921 19678 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 16:42:45.119076 19693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:45.120378 19678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:42:45.120505 19678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "80d51f50278f43acb6547271ab5b8d7b"
format_stamp: "Formatted at 2026-09-27 16:42:45 on dist-test-slave-z40b"
server_key: "68df668d6eed0d352ec80aa0ef75583d"
server_key_iv: "9b8dbe44727231b30631cffadc5ad8fb"
server_key_version: "encryptionkey@0"
I20260927 16:42:45.120611 19678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:45.137904 19678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:45.140584 19678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:45.140769 19678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:45.141305 19678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:42:45.142072 19678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:42:45.142117 19678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:45.142153 19678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:42:45.142204 19678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:45.150555 19678 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43221
I20260927 16:42:45.150588 19806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43221 every 8 connection(s)
I20260927 16:42:45.151459 19678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:42:45.155015 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 19678
I20260927 16:42:45.155179 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:42:45.155552 392 external_mini_cluster.cc:1428] Setting key 42f54ca744c7271f04e2208ac55f7217
Sep 27 16:42:45 dist-test-slave-z40b krb5kdc[19571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 16:42:45.164948 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:45.153155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51013 (local address 127.0.98.62:34571)
0927 16:42:45.153291 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:45.153295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:45.154060 (+ 765us) server_negotiation.cc:407] Connection header received
0927 16:42:45.154973 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:45.154976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:45.155020 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:45.155091 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:45.156470 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.157014 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.158132 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.158529 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.158636 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:45.160977 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:45.160982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:45.160987 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:45.161017 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:45.162802 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:45.163369 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:45.163383 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:45.163385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:45.163476 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:45.163801 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:45.163804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:45.163806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:45.164058 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:45.164170 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:45.164601 (+ 431us) server_negotiation.cc:299] Negotiation successful
0927 16:42:45.164724 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:42:45.165594 19809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:45.153489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34571 (local address 127.0.98.1:51013)
0927 16:42:45.153929 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:45.153968 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:45.154708 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:45.155337 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:45.155347 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:45.155819 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:42:45.156308 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:45.156321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.157141 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.157149 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:45.158009 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:45.158018 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.158686 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.158689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:45.158829 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:45.159405 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:42:45.159420 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:42:45.160821 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:42:45.162910 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:45.162914 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:45.162925 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:45.163163 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:45.163569 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:45.163571 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:45.163572 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:45.163699 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:45.164165 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:42:45.164169 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:42:45.164370 (+ 201us) client_negotiation.cc:769] Sending connection context
0927 16:42:45.164506 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 16:42:45.164731 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":113,"threads_started":1}
I20260927 16:42:45.166747 19807 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34571
I20260927 16:42:45.167012 19807 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 16:42:45.167618 19807 heartbeater.cc:511] Master 127.0.98.62:34571 requested a full tablet report, sending...
I20260927 16:42:45.169239 19616 ts_manager.cc:194] Registered new tserver with Master: 80d51f50278f43acb6547271ab5b8d7b (127.0.98.1:43221)
I20260927 16:42:45.170454 19616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 16:42:45.182111 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34571
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:42:45.284943 19812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:45.285167 19812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:45.285224 19812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:45.289290 19812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:45.289340 19812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:45.289394 19812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:42:45.294273 19812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:45.295176 19812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:45.296144 19812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:45.305423 19817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:45.305557 19812 server_base.cc:1034] running on GCE node
W20260927 16:42:45.305434 19818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:45.305446 19820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:45.306017 19812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:45.306653 19812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:45.307811 19812 hybrid_clock.cc:648] HybridClock initialized: now 1790527365307779 us; error 49 us; skew 500 ppm
Sep 27 16:42:45 dist-test-slave-z40b krb5kdc[19571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:45.310524 19812 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 16:42:45.311322 19812 webserver.cc:460] Webserver started at http://127.0.98.2:36927/ using document root <none> and password file <none>
I20260927 16:42:45.311800 19812 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:45.311852 19812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:45.312005 19812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:45.313846 19812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-1/data/instance:
uuid: "5f34db0a136047e8b81d8dd4dc7b5847"
format_stamp: "Formatted at 2026-09-27 16:42:45 on dist-test-slave-z40b"
server_key: "0046205bd86aa26a43aaeedc8a53ccce"
server_key_iv: "79690916aa6a44db255f1cb347c5f243"
server_key_version: "encryptionkey@0"
I20260927 16:42:45.314335 19812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5f34db0a136047e8b81d8dd4dc7b5847"
format_stamp: "Formatted at 2026-09-27 16:42:45 on dist-test-slave-z40b"
server_key: "0046205bd86aa26a43aaeedc8a53ccce"
server_key_iv: "79690916aa6a44db255f1cb347c5f243"
server_key_version: "encryptionkey@0"
I20260927 16:42:45.317795 19812 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 16:42:45.319977 19827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:45.321060 19812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260927 16:42:45.321193 19812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "5f34db0a136047e8b81d8dd4dc7b5847"
format_stamp: "Formatted at 2026-09-27 16:42:45 on dist-test-slave-z40b"
server_key: "0046205bd86aa26a43aaeedc8a53ccce"
server_key_iv: "79690916aa6a44db255f1cb347c5f243"
server_key_version: "encryptionkey@0"
I20260927 16:42:45.321292 19812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:45.330212 19812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:45.333278 19812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:45.333473 19812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:45.333983 19812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:42:45.334683 19812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:42:45.334728 19812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:45.334789 19812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:42:45.334823 19812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:45.343232 19812 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42467
I20260927 16:42:45.343241 19940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42467 every 8 connection(s)
I20260927 16:42:45.344063 19812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:42:45.347556 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 19812
I20260927 16:42:45.347707 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:42:45.348011 392 external_mini_cluster.cc:1428] Setting key 2a6c0a71f24088406980c4f6a079e6e4
Sep 27 16:42:45 dist-test-slave-z40b krb5kdc[19571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:42:45.357143 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:45.345706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59463 (local address 127.0.98.62:34571)
0927 16:42:45.345828 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:45.345831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:45.346553 (+ 722us) server_negotiation.cc:407] Connection header received
0927 16:42:45.347466 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:45.347469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:45.347529 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:45.347626 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:45.348769 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.349276 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.350330 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.350548 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.350660 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:45.353335 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:45.353341 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:45.353346 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:45.353370 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:45.354908 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:45.355430 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:45.355434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:45.355436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:45.355523 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:45.355917 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:45.355921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:45.355924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:45.356131 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:45.356249 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:45.356707 (+ 458us) server_negotiation.cc:299] Negotiation successful
0927 16:42:45.356858 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 16:42:45.357689 19943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:45.346005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34571 (local address 127.0.98.2:59463)
0927 16:42:45.346413 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:45.346452 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:45.347248 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:45.347803 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:45.347814 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:45.348205 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:42:45.348624 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:45.348636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.349395 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.349399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:45.350208 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:45.350217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.350836 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.350843 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:45.351045 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:45.351615 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:42:45.351638 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:42:45.353184 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:42:45.355013 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:45.355017 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:45.355027 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:45.355334 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:45.355636 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:45.355641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:45.355646 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:45.355811 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:45.356246 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:42:45.356250 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:42:45.356461 (+ 211us) client_negotiation.cc:769] Sending connection context
0927 16:42:45.356599 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 16:42:45.356813 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":105,"threads_started":1}
I20260927 16:42:45.358749 19941 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34571
I20260927 16:42:45.358927 19941 heartbeater.cc:464] Registering TS with master...
I20260927 16:42:45.359412 19941 heartbeater.cc:511] Master 127.0.98.62:34571 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 16:42:45.360349 19616 ts_manager.cc:194] Registered new tserver with Master: 5f34db0a136047e8b81d8dd4dc7b5847 (127.0.98.2:42467)
I20260927 16:42:45.360888 19616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59463
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 16:42:45.374157 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34571
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:42:45.477505 19946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:45.477730 19946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:45.477792 19946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:45.481638 19946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:45.481684 19946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:45.481752 19946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:42:45.486322 19946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:45.487278 19946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:45.488184 19946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:45.496973 19952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:45.496963 19951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:45.497033 19946 server_base.cc:1034] running on GCE node
W20260927 16:42:45.497185 19954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:45.497578 19946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:45.498193 19946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:45.499333 19946 hybrid_clock.cc:648] HybridClock initialized: now 1790527365499321 us; error 24 us; skew 500 ppm
Sep 27 16:42:45 dist-test-slave-z40b krb5kdc[19571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:45.502097 19946 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 16:42:45.502916 19946 webserver.cc:460] Webserver started at http://127.0.98.3:38589/ using document root <none> and password file <none>
I20260927 16:42:45.503472 19946 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:45.503530 19946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:45.503738 19946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:45.505577 19946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-2/data/instance:
uuid: "1e6349155c0b430aa02d423773d50259"
format_stamp: "Formatted at 2026-09-27 16:42:45 on dist-test-slave-z40b"
server_key: "9d918032034b18ed4c60eaea18765909"
server_key_iv: "0b52821b59973d46ae1addf0cf90247e"
server_key_version: "encryptionkey@0"
I20260927 16:42:45.506101 19946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1e6349155c0b430aa02d423773d50259"
format_stamp: "Formatted at 2026-09-27 16:42:45 on dist-test-slave-z40b"
server_key: "9d918032034b18ed4c60eaea18765909"
server_key_iv: "0b52821b59973d46ae1addf0cf90247e"
server_key_version: "encryptionkey@0"
I20260927 16:42:45.509558 19946 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:42:45.511725 19961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:45.512815 19946 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:42:45.512930 19946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "1e6349155c0b430aa02d423773d50259"
format_stamp: "Formatted at 2026-09-27 16:42:45 on dist-test-slave-z40b"
server_key: "9d918032034b18ed4c60eaea18765909"
server_key_iv: "0b52821b59973d46ae1addf0cf90247e"
server_key_version: "encryptionkey@0"
I20260927 16:42:45.513021 19946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:45.530607 19946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:45.533169 19946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:45.533349 19946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:45.533857 19946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:42:45.534584 19946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:42:45.534634 19946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:45.534695 19946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:42:45.534730 19946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:45.543186 19946 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44999
I20260927 16:42:45.543231 20074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44999 every 8 connection(s)
I20260927 16:42:45.544001 19946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:42:45.549420 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 19946
I20260927 16:42:45.549567 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:42:45.549865 392 external_mini_cluster.cc:1428] Setting key b7bbaa18296132c7664ac0c0325c7323
Sep 27 16:42:45 dist-test-slave-z40b krb5kdc[19571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:42:45.556015 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:45.545675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58017 (local address 127.0.98.62:34571)
0927 16:42:45.545812 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:45.545815 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:45.546585 (+ 770us) server_negotiation.cc:407] Connection header received
0927 16:42:45.547409 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:45.547412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:45.547460 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:45.547532 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:45.548600 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.549107 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.550454 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.550773 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.550871 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:45.553229 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:45.553231 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:45.553233 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:45.553254 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:45.554448 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:45.554843 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:45.554845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:45.554846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:45.554912 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:45.555164 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:45.555166 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:45.555167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:45.555369 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:45.555457 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:45.555829 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 16:42:45.555912 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 16:42:45.556862 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:45.546007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34571 (local address 127.0.98.3:58017)
0927 16:42:45.546435 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:45.546474 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:45.547222 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:45.547684 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:45.547690 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:45.548024 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:42:45.548477 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:45.548490 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.549226 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.549228 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:45.550343 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:45.550354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.550903 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.550905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:45.551083 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:45.551725 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:42:45.551740 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:42:45.553104 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:42:45.554539 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:45.554543 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:45.554553 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:45.554766 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:45.554983 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:42:45.554985 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:42:45.554986 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:42:45.555096 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:42:45.555465 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:42:45.555469 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:42:45.555654 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 16:42:45.555787 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 16:42:45.556099 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":100,"threads_started":1}
I20260927 16:42:45.557844 20075 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34571
I20260927 16:42:45.558056 20075 heartbeater.cc:464] Registering TS with master...
I20260927 16:42:45.558485 20075 heartbeater.cc:511] Master 127.0.98.62:34571 requested a full tablet report, sending...
I20260927 16:42:45.559463 19616 ts_manager.cc:194] Registered new tserver with Master: 1e6349155c0b430aa02d423773d50259 (127.0.98.3:44999)
I20260927 16:42:45.559980 19616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58017
I20260927 16:42:45.563336 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:42:45.571394 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:45.565102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47116 (local address 127.0.98.62:34571)
0927 16:42:45.565209 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:45.565212 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:45.565394 (+ 182us) server_negotiation.cc:407] Connection header received
0927 16:42:45.565482 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:45.565483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:45.565515 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:45.565570 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:45.566139 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.566612 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.567453 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.567649 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.567722 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:45.568817 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:45.568820 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:45.568821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:45.568840 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:45.570192 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:45.570515 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:45.570517 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:45.570518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:45.570579 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:45.570771 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:45.570773 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:45.570774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:45.570944 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:45.571035 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:45.571173 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 16:42:45.571305 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 16:42:45.573779 19616 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:42:45.576701 19616 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:42:45.591530 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:45.585564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43221 (local address 127.0.0.1:54802)
0927 16:42:45.586601 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:45.586614 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:45.586689 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:45.587302 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:45.587306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:45.587331 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:45.587423 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:45.587427 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.588788 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.588791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:45.589969 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:45.589977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.590969 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.590973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:45.591103 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:45.591142 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 16:42:45.591283 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 16:42:45.591386 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"mutex_wait_us":26,"thread_start_us":79,"threads_started":1}
I20260927 16:42:45.591603 20091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:45.586323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54802 (local address 127.0.98.1:43221)
0927 16:42:45.586749 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:45.586761 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:45.586782 (+ 21us) server_negotiation.cc:407] Connection header received
0927 16:42:45.586866 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:45.586871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:45.587075 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:45.587258 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:45.587541 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.588615 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.590092 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.590847 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.590957 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:45.591104 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:45.591320 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 16:42:45.591415 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":126,"threads_started":1}
I20260927 16:42:45.591982 20087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:45.585197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44999 (local address 127.0.0.1:60706)
0927 16:42:45.585635 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:45.585678 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:45.585792 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:45.586760 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:45.586767 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:45.586794 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:45.586936 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:45.586942 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.588616 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.588619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:45.590236 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:45.590247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.591565 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.591569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:45.591667 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:45.591697 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 16:42:45.591749 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 16:42:45.591817 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":36,"thread_start_us":61,"threads_started":1}
I20260927 16:42:45.592612 20089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:45.585918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60706 (local address 127.0.98.3:44999)
0927 16:42:45.586226 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:45.586238 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:45.586263 (+ 25us) server_negotiation.cc:407] Connection header received
0927 16:42:45.586335 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:45.586340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:45.586574 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:45.587094 (+ 520us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:45.587315 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.588498 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.590361 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.591431 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.592226 (+ 795us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:45.592354 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:45.592398 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 16:42:45.592476 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":142,"threads_started":1}
I20260927 16:42:45.593582 20088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:45.585304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42467 (local address 127.0.0.1:59110)
0927 16:42:45.586020 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:45.586033 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:45.586113 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:45.589385 (+ 3272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:45.589388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:45.589404 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:45.589507 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:45.589513 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.590711 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.590718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:45.591801 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:45.591812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.593206 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.593210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:45.593312 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:45.593342 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 16:42:45.593414 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 16:42:45.593495 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":94,"threads_started":1}
I20260927 16:42:45.593951 20090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:45.586041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59110 (local address 127.0.98.2:42467)
0927 16:42:45.588812 (+ 2771us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:45.588825 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:45.588846 (+ 21us) server_negotiation.cc:407] Connection header received
0927 16:42:45.588912 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:45.588918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:45.589092 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:45.589233 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:45.589611 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.590596 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.592120 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.592909 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.593027 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:45.593146 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:45.593749 (+ 603us) server_negotiation.cc:299] Negotiation successful
0927 16:42:45.593841 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2666,"thread_start_us":123,"threads_started":1}
I20260927 16:42:45.594844 19741 tablet_service.cc:1467] Processing CreateTablet for tablet fbf7ec4154784e6eb156312e0e79786f (DEFAULT_TABLE table=test-table [id=bbef70a0abde43f588edea9de001ca27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:42:45.595124 20009 tablet_service.cc:1467] Processing CreateTablet for tablet fbf7ec4154784e6eb156312e0e79786f (DEFAULT_TABLE table=test-table [id=bbef70a0abde43f588edea9de001ca27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:42:45.595980 19741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf7ec4154784e6eb156312e0e79786f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:45.596022 20009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf7ec4154784e6eb156312e0e79786f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:45.602578 19875 tablet_service.cc:1467] Processing CreateTablet for tablet fbf7ec4154784e6eb156312e0e79786f (DEFAULT_TABLE table=test-table [id=bbef70a0abde43f588edea9de001ca27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:42:45.602977 20093 tablet_bootstrap.cc:492] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259: Bootstrap starting.
I20260927 16:42:45.603528 20092 tablet_bootstrap.cc:492] T fbf7ec4154784e6eb156312e0e79786f P 80d51f50278f43acb6547271ab5b8d7b: Bootstrap starting.
I20260927 16:42:45.603590 19875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf7ec4154784e6eb156312e0e79786f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:45.605651 20092 tablet_bootstrap.cc:654] T fbf7ec4154784e6eb156312e0e79786f P 80d51f50278f43acb6547271ab5b8d7b: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:45.606882 20092 log.cc:824] T fbf7ec4154784e6eb156312e0e79786f P 80d51f50278f43acb6547271ab5b8d7b: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:45.608561 20092 tablet_bootstrap.cc:492] T fbf7ec4154784e6eb156312e0e79786f P 80d51f50278f43acb6547271ab5b8d7b: No bootstrap required, opened a new log
I20260927 16:42:45.608758 20092 ts_tablet_manager.cc:1397] T fbf7ec4154784e6eb156312e0e79786f P 80d51f50278f43acb6547271ab5b8d7b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260927 16:42:45.609079 20093 tablet_bootstrap.cc:654] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:45.610177 20093 log.cc:824] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:45.610749 20095 tablet_bootstrap.cc:492] T fbf7ec4154784e6eb156312e0e79786f P 5f34db0a136047e8b81d8dd4dc7b5847: Bootstrap starting.
I20260927 16:42:45.612043 20093 tablet_bootstrap.cc:492] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259: No bootstrap required, opened a new log
I20260927 16:42:45.612056 20092 raft_consensus.cc:348] T fbf7ec4154784e6eb156312e0e79786f P 80d51f50278f43acb6547271ab5b8d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6349155c0b430aa02d423773d50259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "5f34db0a136047e8b81d8dd4dc7b5847" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42467 } } peers { permanent_uuid: "80d51f50278f43acb6547271ab5b8d7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43221 } }
I20260927 16:42:45.612241 20093 ts_tablet_manager.cc:1397] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260927 16:42:45.612404 20092 raft_consensus.cc:374] T fbf7ec4154784e6eb156312e0e79786f P 80d51f50278f43acb6547271ab5b8d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:45.612514 20092 raft_consensus.cc:729] T fbf7ec4154784e6eb156312e0e79786f P 80d51f50278f43acb6547271ab5b8d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80d51f50278f43acb6547271ab5b8d7b, State: Initialized, Role: FOLLOWER
I20260927 16:42:45.612746 20095 tablet_bootstrap.cc:654] T fbf7ec4154784e6eb156312e0e79786f P 5f34db0a136047e8b81d8dd4dc7b5847: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:45.613088 20092 consensus_queue.cc:260] T fbf7ec4154784e6eb156312e0e79786f P 80d51f50278f43acb6547271ab5b8d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6349155c0b430aa02d423773d50259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "5f34db0a136047e8b81d8dd4dc7b5847" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42467 } } peers { permanent_uuid: "80d51f50278f43acb6547271ab5b8d7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43221 } }
I20260927 16:42:45.613763 19807 heartbeater.cc:503] Master 127.0.98.62:34571 was elected leader, sending a full tablet report...
I20260927 16:42:45.614079 20092 ts_tablet_manager.cc:1428] T fbf7ec4154784e6eb156312e0e79786f P 80d51f50278f43acb6547271ab5b8d7b: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260927 16:42:45.614470 20095 log.cc:824] T fbf7ec4154784e6eb156312e0e79786f P 5f34db0a136047e8b81d8dd4dc7b5847: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:45.615861 20093 raft_consensus.cc:348] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6349155c0b430aa02d423773d50259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "5f34db0a136047e8b81d8dd4dc7b5847" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42467 } } peers { permanent_uuid: "80d51f50278f43acb6547271ab5b8d7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43221 } }
I20260927 16:42:45.616113 20093 raft_consensus.cc:374] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:45.616205 20093 raft_consensus.cc:729] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e6349155c0b430aa02d423773d50259, State: Initialized, Role: FOLLOWER
I20260927 16:42:45.616673 20093 consensus_queue.cc:260] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6349155c0b430aa02d423773d50259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "5f34db0a136047e8b81d8dd4dc7b5847" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42467 } } peers { permanent_uuid: "80d51f50278f43acb6547271ab5b8d7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43221 } }
I20260927 16:42:45.616882 20095 tablet_bootstrap.cc:492] T fbf7ec4154784e6eb156312e0e79786f P 5f34db0a136047e8b81d8dd4dc7b5847: No bootstrap required, opened a new log
I20260927 16:42:45.617062 20095 ts_tablet_manager.cc:1397] T fbf7ec4154784e6eb156312e0e79786f P 5f34db0a136047e8b81d8dd4dc7b5847: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:42:45.617267 20075 heartbeater.cc:503] Master 127.0.98.62:34571 was elected leader, sending a full tablet report...
I20260927 16:42:45.617594 20093 ts_tablet_manager.cc:1428] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 16:42:45.620416 20095 raft_consensus.cc:348] T fbf7ec4154784e6eb156312e0e79786f P 5f34db0a136047e8b81d8dd4dc7b5847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6349155c0b430aa02d423773d50259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "5f34db0a136047e8b81d8dd4dc7b5847" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42467 } } peers { permanent_uuid: "80d51f50278f43acb6547271ab5b8d7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43221 } }
I20260927 16:42:45.620600 20095 raft_consensus.cc:374] T fbf7ec4154784e6eb156312e0e79786f P 5f34db0a136047e8b81d8dd4dc7b5847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:45.620688 20095 raft_consensus.cc:729] T fbf7ec4154784e6eb156312e0e79786f P 5f34db0a136047e8b81d8dd4dc7b5847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f34db0a136047e8b81d8dd4dc7b5847, State: Initialized, Role: FOLLOWER
I20260927 16:42:45.621109 20095 consensus_queue.cc:260] T fbf7ec4154784e6eb156312e0e79786f P 5f34db0a136047e8b81d8dd4dc7b5847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6349155c0b430aa02d423773d50259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "5f34db0a136047e8b81d8dd4dc7b5847" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42467 } } peers { permanent_uuid: "80d51f50278f43acb6547271ab5b8d7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43221 } }
I20260927 16:42:45.621678 19941 heartbeater.cc:503] Master 127.0.98.62:34571 was elected leader, sending a full tablet report...
I20260927 16:42:45.621892 20095 ts_tablet_manager.cc:1428] T fbf7ec4154784e6eb156312e0e79786f P 5f34db0a136047e8b81d8dd4dc7b5847: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:42:45.622997 20099 raft_consensus.cc:482] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:42:45.623152 20099 raft_consensus.cc:504] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6349155c0b430aa02d423773d50259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "5f34db0a136047e8b81d8dd4dc7b5847" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42467 } } peers { permanent_uuid: "80d51f50278f43acb6547271ab5b8d7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43221 } }
I20260927 16:42:45.624332 20099 leader_election.cc:288] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f34db0a136047e8b81d8dd4dc7b5847 (127.0.98.2:42467), 80d51f50278f43acb6547271ab5b8d7b (127.0.98.1:43221)
I20260927 16:42:45.626837 20100 raft_consensus.cc:482] T fbf7ec4154784e6eb156312e0e79786f P 5f34db0a136047e8b81d8dd4dc7b5847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:42:45.626972 20100 raft_consensus.cc:504] T fbf7ec4154784e6eb156312e0e79786f P 5f34db0a136047e8b81d8dd4dc7b5847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6349155c0b430aa02d423773d50259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "5f34db0a136047e8b81d8dd4dc7b5847" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42467 } } peers { permanent_uuid: "80d51f50278f43acb6547271ab5b8d7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43221 } }
I20260927 16:42:45.628201 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:45.624484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42467 (local address 127.0.98.3:60131)
0927 16:42:45.624598 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:45.624613 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:45.624702 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:45.625155 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:45.625158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:45.625191 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:45.625310 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:45.625314 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.626132 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.626134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:45.626895 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:45.626902 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.627788 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.627792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:45.627893 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:45.627914 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 16:42:45.628006 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 16:42:45.628079 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 16:42:45.628199 20090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:45.624737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60131 (local address 127.0.98.2:42467)
0927 16:42:45.624854 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:45.624857 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:45.624870 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:42:45.624908 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:45.624911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:45.624949 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:45.625038 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:45.625403 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.626023 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.627064 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.627669 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.627790 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:45.627846 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:45.628032 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 16:42:45.628094 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 16:42:45.628705 19895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf7ec4154784e6eb156312e0e79786f" candidate_uuid: "1e6349155c0b430aa02d423773d50259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f34db0a136047e8b81d8dd4dc7b5847" is_pre_election: true
I20260927 16:42:45.628887 19895 raft_consensus.cc:2454] T fbf7ec4154784e6eb156312e0e79786f P 5f34db0a136047e8b81d8dd4dc7b5847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e6349155c0b430aa02d423773d50259 in term 0.
I20260927 16:42:45.629384 20091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:45.624983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40707 (local address 127.0.98.1:43221)
0927 16:42:45.625126 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:45.625130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:45.625662 (+ 532us) server_negotiation.cc:407] Connection header received
0927 16:42:45.625735 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:45.625738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:45.625777 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:45.625844 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:45.626094 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.626633 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.628100 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.628668 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.628749 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:45.628801 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:45.629187 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 16:42:45.629259 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 16:42:45.629387 19964 leader_election.cc:302] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e6349155c0b430aa02d423773d50259, 5f34db0a136047e8b81d8dd4dc7b5847; no voters:
I20260927 16:42:45.629665 20099 raft_consensus.cc:2781] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:42:45.629716 20100 leader_election.cc:288] T fbf7ec4154784e6eb156312e0e79786f P 5f34db0a136047e8b81d8dd4dc7b5847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e6349155c0b430aa02d423773d50259 (127.0.98.3:44999), 80d51f50278f43acb6547271ab5b8d7b (127.0.98.1:43221)
I20260927 16:42:45.629745 20099 raft_consensus.cc:482] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:42:45.629788 20099 raft_consensus.cc:3037] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:45.630041 20101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:45.624667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43221 (local address 127.0.98.3:40707)
0927 16:42:45.625564 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:45.625577 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:45.625660 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:45.625860 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:45.625862 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:45.625873 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:45.625973 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:45.625977 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.626749 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.626754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:45.627962 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:45.627974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.628908 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.628912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:45.629036 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:45.629069 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 16:42:45.629884 (+ 815us) client_negotiation.cc:240] Negotiation successful
0927 16:42:45.629945 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"thread_start_us":83,"threads_started":1}
I20260927 16:42:45.630966 20099 raft_consensus.cc:504] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6349155c0b430aa02d423773d50259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "5f34db0a136047e8b81d8dd4dc7b5847" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42467 } } peers { permanent_uuid: "80d51f50278f43acb6547271ab5b8d7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43221 } }
I20260927 16:42:45.631300 20099 leader_election.cc:288] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259 [CANDIDATE]: Term 1 election: Requested vote from peers 5f34db0a136047e8b81d8dd4dc7b5847 (127.0.98.2:42467), 80d51f50278f43acb6547271ab5b8d7b (127.0.98.1:43221)
I20260927 16:42:45.631560 19895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf7ec4154784e6eb156312e0e79786f" candidate_uuid: "1e6349155c0b430aa02d423773d50259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f34db0a136047e8b81d8dd4dc7b5847"
I20260927 16:42:45.631579 19761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf7ec4154784e6eb156312e0e79786f" candidate_uuid: "1e6349155c0b430aa02d423773d50259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80d51f50278f43acb6547271ab5b8d7b" is_pre_election: true
I20260927 16:42:45.631669 19895 raft_consensus.cc:3037] T fbf7ec4154784e6eb156312e0e79786f P 5f34db0a136047e8b81d8dd4dc7b5847 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:45.631824 19761 raft_consensus.cc:2454] T fbf7ec4154784e6eb156312e0e79786f P 80d51f50278f43acb6547271ab5b8d7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e6349155c0b430aa02d423773d50259 in term 0.
I20260927 16:42:45.632681 19761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf7ec4154784e6eb156312e0e79786f" candidate_uuid: "1e6349155c0b430aa02d423773d50259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80d51f50278f43acb6547271ab5b8d7b"
I20260927 16:42:45.632761 19895 raft_consensus.cc:2454] T fbf7ec4154784e6eb156312e0e79786f P 5f34db0a136047e8b81d8dd4dc7b5847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e6349155c0b430aa02d423773d50259 in term 1.
I20260927 16:42:45.632802 19761 raft_consensus.cc:3037] T fbf7ec4154784e6eb156312e0e79786f P 80d51f50278f43acb6547271ab5b8d7b [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:45.632975 19943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:45.628797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44999 (local address 127.0.98.2:46829)
0927 16:42:45.629034 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:45.629047 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:45.629115 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:45.629377 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:45.629381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:45.629401 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:45.629520 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:45.629526 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.630417 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.630420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:45.631637 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:45.631647 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.632562 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.632567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:45.632676 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:45.632708 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 16:42:45.632780 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 16:42:45.632856 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":26}
I20260927 16:42:45.633198 19964 leader_election.cc:302] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e6349155c0b430aa02d423773d50259, 5f34db0a136047e8b81d8dd4dc7b5847; no voters:
I20260927 16:42:45.633338 20099 raft_consensus.cc:2781] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:42:45.633474 20089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:45.629027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46829 (local address 127.0.98.3:44999)
0927 16:42:45.629126 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:45.629132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:45.629146 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:42:45.629185 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:45.629189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:45.629234 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:45.629294 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:45.629644 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.630292 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.631807 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.632347 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.632439 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:45.632500 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:45.633244 (+ 744us) server_negotiation.cc:299] Negotiation successful
0927 16:42:45.633321 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 16:42:45.633518 20099 raft_consensus.cc:686] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259 [term 1 LEADER]: Becoming Leader. State: Replica: 1e6349155c0b430aa02d423773d50259, State: Running, Role: LEADER
I20260927 16:42:45.633821 20099 consensus_queue.cc:237] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6349155c0b430aa02d423773d50259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "5f34db0a136047e8b81d8dd4dc7b5847" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42467 } } peers { permanent_uuid: "80d51f50278f43acb6547271ab5b8d7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43221 } }
I20260927 16:42:45.633860 20029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf7ec4154784e6eb156312e0e79786f" candidate_uuid: "5f34db0a136047e8b81d8dd4dc7b5847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e6349155c0b430aa02d423773d50259" is_pre_election: true
I20260927 16:42:45.634038 19761 raft_consensus.cc:2454] T fbf7ec4154784e6eb156312e0e79786f P 80d51f50278f43acb6547271ab5b8d7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e6349155c0b430aa02d423773d50259 in term 1.
I20260927 16:42:45.634162 20091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:45.629755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47193 (local address 127.0.98.1:43221)
0927 16:42:45.629849 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:45.629853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:45.630367 (+ 514us) server_negotiation.cc:407] Connection header received
0927 16:42:45.630526 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:45.630529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:45.630572 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:45.630649 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:45.630897 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.631460 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.632924 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.633518 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.633597 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:45.633646 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:45.634005 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 16:42:45.634067 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 16:42:45.634531 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:45.629448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43221 (local address 127.0.98.2:47193)
0927 16:42:45.630294 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:45.630307 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:45.630427 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:45.630659 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:45.630662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:45.630677 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:45.630800 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:45.630805 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.631579 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.631584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:45.632782 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:45.632793 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.633760 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.633763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:45.633861 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:45.633884 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 16:42:45.634316 (+ 432us) client_negotiation.cc:240] Negotiation successful
0927 16:42:45.634385 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":1,"thread_start_us":108,"threads_started":1}
I20260927 16:42:45.634932 19761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf7ec4154784e6eb156312e0e79786f" candidate_uuid: "5f34db0a136047e8b81d8dd4dc7b5847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80d51f50278f43acb6547271ab5b8d7b" is_pre_election: true
I20260927 16:42:45.635093 19761 raft_consensus.cc:2379] T fbf7ec4154784e6eb156312e0e79786f P 80d51f50278f43acb6547271ab5b8d7b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f34db0a136047e8b81d8dd4dc7b5847 in current term 1: Already voted for candidate 1e6349155c0b430aa02d423773d50259 in this term.
I20260927 16:42:45.635522 19829 leader_election.cc:302] T fbf7ec4154784e6eb156312e0e79786f P 5f34db0a136047e8b81d8dd4dc7b5847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f34db0a136047e8b81d8dd4dc7b5847; no voters: 1e6349155c0b430aa02d423773d50259, 80d51f50278f43acb6547271ab5b8d7b
I20260927 16:42:45.635741 20100 raft_consensus.cc:2726] T fbf7ec4154784e6eb156312e0e79786f P 5f34db0a136047e8b81d8dd4dc7b5847 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 16:42:45.636600 19616 catalog_manager.cc:5495] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e6349155c0b430aa02d423773d50259 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1e6349155c0b430aa02d423773d50259" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6349155c0b430aa02d423773d50259" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f34db0a136047e8b81d8dd4dc7b5847" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80d51f50278f43acb6547271ab5b8d7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43221 } health_report { overall_health: UNKNOWN } } }
W20260927 16:42:45.652539 19808 tablet.cc:2369] T fbf7ec4154784e6eb156312e0e79786f P 80d51f50278f43acb6547271ab5b8d7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:42:45.658465 20089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:45.654966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60714 (local address 127.0.98.3:44999)
0927 16:42:45.655089 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:45.655096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:45.655114 (+ 18us) server_negotiation.cc:407] Connection header received
0927 16:42:45.655336 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:45.655338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:45.655389 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:45.655496 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:42:45.655738 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.656230 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.657096 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.657307 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.657408 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:45.657722 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:42:45.658102 (+ 380us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:42:45.658232 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:45.658277 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 16:42:45.658326 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:42:45.663839 19761 raft_consensus.cc:1261] T fbf7ec4154784e6eb156312e0e79786f P 80d51f50278f43acb6547271ab5b8d7b [term 1 FOLLOWER]: Refusing update from remote peer 1e6349155c0b430aa02d423773d50259: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:42:45.663877 19895 raft_consensus.cc:1261] T fbf7ec4154784e6eb156312e0e79786f P 5f34db0a136047e8b81d8dd4dc7b5847 [term 1 FOLLOWER]: Refusing update from remote peer 1e6349155c0b430aa02d423773d50259: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:42:45.664450 20103 consensus_queue.cc:1035] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f34db0a136047e8b81d8dd4dc7b5847" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:42:45.664721 20099 consensus_queue.cc:1035] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d51f50278f43acb6547271ab5b8d7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:42:45.672277 20107 mvcc.cc:205] Tried to move back new op lower bound from 7334000089752444928 to 7334000089638928384. Current Snapshot: MvccSnapshot[applied={T|T < 7334000089752444928}]
I20260927 16:42:45.679831 19616 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:42:45.679993 19616 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:42:45.682025 19616 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42e2e55b05bd44ed848a9843e153f665: Sending DeleteTablet for 3 replicas of tablet fbf7ec4154784e6eb156312e0e79786f
I20260927 16:42:45.682705 19741 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf7ec4154784e6eb156312e0e79786f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:42:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54802
I20260927 16:42:45.682705 20009 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf7ec4154784e6eb156312e0e79786f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:42:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60706
I20260927 16:42:45.682717 19875 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf7ec4154784e6eb156312e0e79786f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:42:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59110
I20260927 16:42:45.683118 20116 tablet_replica.cc:330] T fbf7ec4154784e6eb156312e0e79786f P 80d51f50278f43acb6547271ab5b8d7b: stopping tablet replica
I20260927 16:42:45.683388 20116 raft_consensus.cc:2229] T fbf7ec4154784e6eb156312e0e79786f P 80d51f50278f43acb6547271ab5b8d7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:42:45.683672 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 19678
I20260927 16:42:45.683663 20116 raft_consensus.cc:2258] T fbf7ec4154784e6eb156312e0e79786f P 80d51f50278f43acb6547271ab5b8d7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:42:45.687314 20118 tablet_replica.cc:330] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259: stopping tablet replica
I20260927 16:42:45.687561 20118 raft_consensus.cc:2229] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:42:45.687870 20118 raft_consensus.cc:2258] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:42:45.688223 20117 tablet_replica.cc:330] T fbf7ec4154784e6eb156312e0e79786f P 5f34db0a136047e8b81d8dd4dc7b5847: stopping tablet replica
I20260927 16:42:45.688447 20117 raft_consensus.cc:2229] T fbf7ec4154784e6eb156312e0e79786f P 5f34db0a136047e8b81d8dd4dc7b5847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:42:45.688679 20117 raft_consensus.cc:2258] T fbf7ec4154784e6eb156312e0e79786f P 5f34db0a136047e8b81d8dd4dc7b5847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:42:45.689803 20118 ts_tablet_manager.cc:1905] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 16:42:45.691298 19601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43221 (error 108)
I20260927 16:42:45.691534 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 19812
I20260927 16:42:45.692370 20088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:45.692020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43221 (local address 127.0.0.1:54806)
0927 16:42:45.692203 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:45.692275 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260927 16:42:45.692534 19601 catalog_manager.cc:4584] TS 80d51f50278f43acb6547271ab5b8d7b (127.0.98.1:43221): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fbf7ec4154784e6eb156312e0e79786f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43221: connect: Connection refused (error 111)
I20260927 16:42:45.693023 20118 ts_tablet_manager.cc:1918] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:42:45.693123 20118 log.cc:1196] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790527121515808-392-0/minicluster-data/ts-2/wal/wals/fbf7ec4154784e6eb156312e0e79786f
I20260927 16:42:45.693410 20118 ts_tablet_manager.cc:1939] T fbf7ec4154784e6eb156312e0e79786f P 1e6349155c0b430aa02d423773d50259: Deleting consensus metadata
I20260927 16:42:45.694965 19603 catalog_manager.cc:4841] TS 1e6349155c0b430aa02d423773d50259 (127.0.98.3:44999): tablet fbf7ec4154784e6eb156312e0e79786f (table test-table [id=bbef70a0abde43f588edea9de001ca27]) successfully deleted
I20260927 16:42:45.698335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 19946
I20260927 16:42:45.698748 20088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:45.698472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42467 (local address 127.0.0.1:59120)
0927 16:42:45.698586 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:45.698663 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 16:42:45.703874 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 19584
2026-09-27T16:42:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2174 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T16:42:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:42:45Z Disabled control of system clock
I20260927 16:42:45.732722 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37911
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 16:42:45.832442 20123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:45.832664 20123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:45.832696 20123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:45.836501 20123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:42:45.836545 20123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:45.836560 20123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:45.836576 20123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:42:45.836591 20123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:42:45.841281 20123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37911
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:45.842187 20123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:45.843418 20123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:45.851130 20131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:45.851241 20129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:45.851377 20123 server_base.cc:1034] running on GCE node
W20260927 16:42:45.851240 20128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:45.852097 20123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:45.853065 20123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:45.854218 20123 hybrid_clock.cc:648] HybridClock initialized: now 1790527365854194 us; error 40 us; skew 500 ppm
I20260927 16:42:45.856007 20123 webserver.cc:460] Webserver started at http://127.0.98.62:46353/ using document root <none> and password file <none>
I20260927 16:42:45.856519 20123 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:45.856578 20123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:45.856730 20123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:45.858651 20123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "0f3a4966e21f4c22a40d8dc21dd18f53"
format_stamp: "Formatted at 2026-09-27 16:42:45 on dist-test-slave-z40b"
server_key: "30166d0fa741e8e942018ee104ebb959"
server_key_iv: "850d81f126e121c0409e4ffdf226e484"
server_key_version: "encryptionkey@0"
I20260927 16:42:45.859133 20123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "0f3a4966e21f4c22a40d8dc21dd18f53"
format_stamp: "Formatted at 2026-09-27 16:42:45 on dist-test-slave-z40b"
server_key: "30166d0fa741e8e942018ee104ebb959"
server_key_iv: "850d81f126e121c0409e4ffdf226e484"
server_key_version: "encryptionkey@0"
I20260927 16:42:45.862588 20123 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:42:45.864811 20137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:45.865923 20123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:42:45.866067 20123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "0f3a4966e21f4c22a40d8dc21dd18f53"
format_stamp: "Formatted at 2026-09-27 16:42:45 on dist-test-slave-z40b"
server_key: "30166d0fa741e8e942018ee104ebb959"
server_key_iv: "850d81f126e121c0409e4ffdf226e484"
server_key_version: "encryptionkey@0"
I20260927 16:42:45.866168 20123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:45.885566 20123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:45.886153 20123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:45.886322 20123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:45.892756 20123 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37911
I20260927 16:42:45.892761 20189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37911 every 8 connection(s)
I20260927 16:42:45.893587 20123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:42:45.896518 20190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:45.897723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 20123
I20260927 16:42:45.897874 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:42:45.898173 392 external_mini_cluster.cc:1428] Setting key 1a3c47258d6bc2c3682ba4cb2ec19373
I20260927 16:42:45.902174 20190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f3a4966e21f4c22a40d8dc21dd18f53: Bootstrap starting.
I20260927 16:42:45.904249 20190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f3a4966e21f4c22a40d8dc21dd18f53: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:45.905167 20190 log.cc:824] T 00000000000000000000000000000000 P 0f3a4966e21f4c22a40d8dc21dd18f53: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:45.907076 20190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f3a4966e21f4c22a40d8dc21dd18f53: No bootstrap required, opened a new log
I20260927 16:42:45.907181 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:45.899464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38464 (local address 127.0.98.62:37911)
0927 16:42:45.900085 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:45.900107 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:45.900153 (+ 46us) server_negotiation.cc:407] Connection header received
0927 16:42:45.900772 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:45.900788 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:45.901271 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:45.901642 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:45.902046 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.903012 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.903920 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:45.904338 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:45.904524 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:45.905389 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:45.905394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:45.905405 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:42:45.905883 (+ 478us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:45.905949 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:42:45.905969 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:45.906054 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:45.906204 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 16:42:45.906487 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":129,"threads_started":1}
I20260927 16:42:45.910727 20190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f3a4966e21f4c22a40d8dc21dd18f53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3a4966e21f4c22a40d8dc21dd18f53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37911 } }
I20260927 16:42:45.910918 20190 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f3a4966e21f4c22a40d8dc21dd18f53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:45.911002 20190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f3a4966e21f4c22a40d8dc21dd18f53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3a4966e21f4c22a40d8dc21dd18f53, State: Initialized, Role: FOLLOWER
I20260927 16:42:45.911423 20190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f3a4966e21f4c22a40d8dc21dd18f53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3a4966e21f4c22a40d8dc21dd18f53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37911 } }
I20260927 16:42:45.911603 20190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f3a4966e21f4c22a40d8dc21dd18f53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:42:45.911661 20190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f3a4966e21f4c22a40d8dc21dd18f53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:42:45.911748 20190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f3a4966e21f4c22a40d8dc21dd18f53 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:45.912552 20190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f3a4966e21f4c22a40d8dc21dd18f53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3a4966e21f4c22a40d8dc21dd18f53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37911 } }
I20260927 16:42:45.912859 20190 leader_election.cc:302] T 00000000000000000000000000000000 P 0f3a4966e21f4c22a40d8dc21dd18f53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f3a4966e21f4c22a40d8dc21dd18f53; no voters:
I20260927 16:42:45.913195 20190 leader_election.cc:288] T 00000000000000000000000000000000 P 0f3a4966e21f4c22a40d8dc21dd18f53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:42:45.913301 20195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f3a4966e21f4c22a40d8dc21dd18f53 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:42:45.913527 20195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f3a4966e21f4c22a40d8dc21dd18f53 [term 1 LEADER]: Becoming Leader. State: Replica: 0f3a4966e21f4c22a40d8dc21dd18f53, State: Running, Role: LEADER
I20260927 16:42:45.913841 20195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f3a4966e21f4c22a40d8dc21dd18f53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3a4966e21f4c22a40d8dc21dd18f53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37911 } }
I20260927 16:42:45.914288 20190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f3a4966e21f4c22a40d8dc21dd18f53 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:42:45.915294 20197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f3a4966e21f4c22a40d8dc21dd18f53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f3a4966e21f4c22a40d8dc21dd18f53. Latest consensus state: current_term: 1 leader_uuid: "0f3a4966e21f4c22a40d8dc21dd18f53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3a4966e21f4c22a40d8dc21dd18f53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37911 } } }
I20260927 16:42:45.915429 20197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f3a4966e21f4c22a40d8dc21dd18f53 [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:45.916021 20196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f3a4966e21f4c22a40d8dc21dd18f53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f3a4966e21f4c22a40d8dc21dd18f53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3a4966e21f4c22a40d8dc21dd18f53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37911 } } }
I20260927 16:42:45.916240 20196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f3a4966e21f4c22a40d8dc21dd18f53 [sys.catalog]: This master's current role is: LEADER
W20260927 16:42:45.918752 20211 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f3a4966e21f4c22a40d8dc21dd18f53: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:42:45.918836 20211 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:42:45.918913 20204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:42:45.919673 20204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:42:45.924808 20204 catalog_manager.cc:1347] Generated new cluster ID: 4733605a2b094360bec29a20dc6a49c0
I20260927 16:42:45.924914 20204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:42:45.957482 20204 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:42:45.958334 20204 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:42:45.970144 20204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f3a4966e21f4c22a40d8dc21dd18f53: Generated new TSK 0
I20260927 16:42:45.970834 20204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:42:45.977051 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37911
--builtin_ntp_servers=127.0.98.20:42469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 16:42:46.081223 20214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:46.081444 20214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:46.081478 20214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:46.085152 20214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:46.085192 20214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:46.085258 20214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 16:42:46.090026 20214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:46.090972 20214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:46.092274 20214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:46.101365 20219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:46.101373 20222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:46.101385 20220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:46.101684 20214 server_base.cc:1034] running on GCE node
I20260927 16:42:46.102080 20214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:46.102715 20214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:46.103860 20214 hybrid_clock.cc:648] HybridClock initialized: now 1790527366103832 us; error 40 us; skew 500 ppm
I20260927 16:42:46.105648 20214 webserver.cc:460] Webserver started at http://127.0.98.1:42417/ using document root <none> and password file <none>
I20260927 16:42:46.106175 20214 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:46.106232 20214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:46.106470 20214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:46.108566 20214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-0/data/instance:
uuid: "53ea075b7e23435fbb3b7fb81e0b0252"
format_stamp: "Formatted at 2026-09-27 16:42:46 on dist-test-slave-z40b"
server_key: "353d15af1766fcf1087370a43409ea33"
server_key_iv: "2a14df80d18e49fcbdd99595bc6ef884"
server_key_version: "encryptionkey@0"
I20260927 16:42:46.109128 20214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-0/wal/instance:
uuid: "53ea075b7e23435fbb3b7fb81e0b0252"
format_stamp: "Formatted at 2026-09-27 16:42:46 on dist-test-slave-z40b"
server_key: "353d15af1766fcf1087370a43409ea33"
server_key_iv: "2a14df80d18e49fcbdd99595bc6ef884"
server_key_version: "encryptionkey@0"
I20260927 16:42:46.112651 20214 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:42:46.114972 20228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:46.116255 20214 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:42:46.116364 20214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-0/wal
uuid: "53ea075b7e23435fbb3b7fb81e0b0252"
format_stamp: "Formatted at 2026-09-27 16:42:46 on dist-test-slave-z40b"
server_key: "353d15af1766fcf1087370a43409ea33"
server_key_iv: "2a14df80d18e49fcbdd99595bc6ef884"
server_key_version: "encryptionkey@0"
I20260927 16:42:46.116472 20214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:46.139622 20214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:46.140187 20214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:46.140362 20214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:46.140916 20214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:42:46.141760 20214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:42:46.141804 20214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:46.141868 20214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:42:46.141898 20214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:46.150072 20214 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45429
I20260927 16:42:46.150095 20341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45429 every 8 connection(s)
I20260927 16:42:46.150835 20214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-0/data/info.pb
I20260927 16:42:46.152946 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 20214
I20260927 16:42:46.153088 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-0/wal/instance
I20260927 16:42:46.153388 392 external_mini_cluster.cc:1428] Setting key 1f173f853d4cd6db22595a8e1e23c019
I20260927 16:42:46.155457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37911
--builtin_ntp_servers=127.0.98.20:42469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 16:42:46.159989 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:46.152428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34829 (local address 127.0.98.62:37911)
0927 16:42:46.152586 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:46.152590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:46.153620 (+ 1030us) server_negotiation.cc:407] Connection header received
0927 16:42:46.154925 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:46.154929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:46.154982 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:46.155077 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:46.156019 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.156884 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.158036 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.158264 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.158375 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:46.159508 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:46.159511 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:46.159514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:42:46.159525 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:46.159549 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:42:46.159556 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:46.159627 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:46.159787 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 16:42:46.159849 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:42:46.160722 20344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:46.152805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37911 (local address 127.0.98.1:34829)
0927 16:42:46.153463 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:46.153511 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:46.154739 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:46.155281 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:46.155288 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:46.155367 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:42:46.155877 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:46.155891 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.157011 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.157015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:46.157906 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:46.157916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.158417 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.158422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:46.158588 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:46.159197 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:42:46.159199 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:42:46.159272 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:42:46.159342 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:42:46.159641 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:42:46.159651 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:42:46.159657 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 16:42:46.159785 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 16:42:46.160030 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":133,"threads_started":1}
I20260927 16:42:46.161967 20342 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37911
I20260927 16:42:46.162180 20342 heartbeater.cc:464] Registering TS with master...
I20260927 16:42:46.162623 20342 heartbeater.cc:511] Master 127.0.98.62:37911 requested a full tablet report, sending...
I20260927 16:42:46.164201 20154 ts_manager.cc:194] Registered new tserver with Master: 53ea075b7e23435fbb3b7fb81e0b0252 (127.0.98.1:45429)
I20260927 16:42:46.165772 20154 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:34829
W20260927 16:42:46.260777 20345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:46.260993 20345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:46.261029 20345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:46.264752 20345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:46.264792 20345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:46.264856 20345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 16:42:46.269193 20345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:46.270126 20345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:46.271457 20345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:46.280071 20350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:46.280153 20351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:46.280139 20353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:46.280254 20345 server_base.cc:1034] running on GCE node
I20260927 16:42:46.280679 20345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:46.281263 20345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:46.282409 20345 hybrid_clock.cc:648] HybridClock initialized: now 1790527366282396 us; error 23 us; skew 500 ppm
I20260927 16:42:46.284119 20345 webserver.cc:460] Webserver started at http://127.0.98.2:34683/ using document root <none> and password file <none>
I20260927 16:42:46.284631 20345 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:46.284684 20345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:46.284919 20345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:46.286778 20345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-1/data/instance:
uuid: "b62d589cfc2f45b58b7c464d09125841"
format_stamp: "Formatted at 2026-09-27 16:42:46 on dist-test-slave-z40b"
server_key: "c36f7cb9d36e3a1e1c09a2c81d029aee"
server_key_iv: "d03877b9f1b0025b912ae4752a9de426"
server_key_version: "encryptionkey@0"
I20260927 16:42:46.287385 20345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b62d589cfc2f45b58b7c464d09125841"
format_stamp: "Formatted at 2026-09-27 16:42:46 on dist-test-slave-z40b"
server_key: "c36f7cb9d36e3a1e1c09a2c81d029aee"
server_key_iv: "d03877b9f1b0025b912ae4752a9de426"
server_key_version: "encryptionkey@0"
I20260927 16:42:46.290678 20345 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:42:46.292766 20359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:46.293784 20345 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 16:42:46.293891 20345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-1/wal
uuid: "b62d589cfc2f45b58b7c464d09125841"
format_stamp: "Formatted at 2026-09-27 16:42:46 on dist-test-slave-z40b"
server_key: "c36f7cb9d36e3a1e1c09a2c81d029aee"
server_key_iv: "d03877b9f1b0025b912ae4752a9de426"
server_key_version: "encryptionkey@0"
I20260927 16:42:46.293988 20345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:46.332321 20345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:46.332998 20345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:46.333209 20345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:46.333813 20345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:42:46.334734 20345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:42:46.334779 20345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:46.334848 20345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:42:46.334892 20345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:46.343755 20345 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37851
I20260927 16:42:46.343786 20472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37851 every 8 connection(s)
I20260927 16:42:46.344587 20345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-1/data/info.pb
I20260927 16:42:46.351576 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 20345
I20260927 16:42:46.351722 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-1/wal/instance
I20260927 16:42:46.351991 392 external_mini_cluster.cc:1428] Setting key e9455693f9441034362388e23728b0c4
I20260927 16:42:46.353593 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:46.346311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48277 (local address 127.0.98.62:37911)
0927 16:42:46.346447 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:46.346451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:46.347442 (+ 991us) server_negotiation.cc:407] Connection header received
0927 16:42:46.348859 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:46.348863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:46.348924 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:46.349036 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:46.349885 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.350416 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.351629 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.351940 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.352044 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:46.353091 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:46.353094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:46.353096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:42:46.353108 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:46.353133 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:42:46.353141 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:46.353230 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:46.353399 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 16:42:46.353462 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:42:46.354189 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37911
--builtin_ntp_servers=127.0.98.20:42469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 16:42:46.354401 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:46.346633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37911 (local address 127.0.98.2:48277)
0927 16:42:46.347276 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:46.347314 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:46.348606 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:46.349195 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:46.349201 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:46.349278 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:42:46.349718 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:46.349729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.350552 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.350555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:46.351509 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:46.351518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.352081 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.352085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:46.352236 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:46.352837 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:42:46.352838 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:42:46.352875 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:42:46.352943 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:42:46.353242 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:42:46.353252 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:42:46.353259 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 16:42:46.353392 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 16:42:46.353644 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":133,"threads_started":1}
I20260927 16:42:46.355466 20473 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37911
I20260927 16:42:46.355664 20473 heartbeater.cc:464] Registering TS with master...
I20260927 16:42:46.356135 20473 heartbeater.cc:511] Master 127.0.98.62:37911 requested a full tablet report, sending...
I20260927 16:42:46.357136 20154 ts_manager.cc:194] Registered new tserver with Master: b62d589cfc2f45b58b7c464d09125841 (127.0.98.2:37851)
I20260927 16:42:46.357987 20154 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:48277
W20260927 16:42:46.459681 20476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:46.459897 20476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:46.459931 20476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:46.464309 20476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:46.464356 20476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:46.464408 20476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 16:42:46.468876 20476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:46.469781 20476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:46.471268 20476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:46.480840 20481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:46.480840 20484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:46.481050 20482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:46.482151 20476 server_base.cc:1034] running on GCE node
I20260927 16:42:46.482517 20476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:46.483166 20476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:46.484361 20476 hybrid_clock.cc:648] HybridClock initialized: now 1790527366484350 us; error 49 us; skew 500 ppm
I20260927 16:42:46.486377 20476 webserver.cc:460] Webserver started at http://127.0.98.3:42819/ using document root <none> and password file <none>
I20260927 16:42:46.487003 20476 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:46.487068 20476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:46.487358 20476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:46.489306 20476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-2/data/instance:
uuid: "c2a7ecb4cfd9401d97c4dc1ff862386b"
format_stamp: "Formatted at 2026-09-27 16:42:46 on dist-test-slave-z40b"
server_key: "f2c18b4f491741c7ae9cbbcc985b341a"
server_key_iv: "7e867bb650d35df6d38957ab91bd8f35"
server_key_version: "encryptionkey@0"
I20260927 16:42:46.489840 20476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c2a7ecb4cfd9401d97c4dc1ff862386b"
format_stamp: "Formatted at 2026-09-27 16:42:46 on dist-test-slave-z40b"
server_key: "f2c18b4f491741c7ae9cbbcc985b341a"
server_key_iv: "7e867bb650d35df6d38957ab91bd8f35"
server_key_version: "encryptionkey@0"
I20260927 16:42:46.493309 20476 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20260927 16:42:46.495534 20490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:46.496634 20476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:42:46.496745 20476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-2/wal
uuid: "c2a7ecb4cfd9401d97c4dc1ff862386b"
format_stamp: "Formatted at 2026-09-27 16:42:46 on dist-test-slave-z40b"
server_key: "f2c18b4f491741c7ae9cbbcc985b341a"
server_key_iv: "7e867bb650d35df6d38957ab91bd8f35"
server_key_version: "encryptionkey@0"
I20260927 16:42:46.496840 20476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:46.522089 20476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:46.522751 20476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:46.522933 20476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:46.523583 20476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:42:46.524510 20476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:42:46.524569 20476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:46.524631 20476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:42:46.524669 20476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:46.533242 20476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39049
I20260927 16:42:46.533315 20603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39049 every 8 connection(s)
I20260927 16:42:46.534121 20476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-2/data/info.pb
I20260927 16:42:46.540305 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 20476
I20260927 16:42:46.540416 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-2/wal/instance
I20260927 16:42:46.540736 392 external_mini_cluster.cc:1428] Setting key d8eba165633d6bed84b691e6b2711e30
I20260927 16:42:46.543233 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:46.535875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36365 (local address 127.0.98.62:37911)
0927 16:42:46.536018 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:46.536022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:46.537030 (+ 1008us) server_negotiation.cc:407] Connection header received
0927 16:42:46.538303 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:46.538306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:46.538350 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:46.538425 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:46.539292 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.539787 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.541281 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.541590 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.541704 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:46.542728 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:46.542731 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:46.542734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:42:46.542745 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:46.542779 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:42:46.542788 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:46.542860 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:46.543025 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 16:42:46.543083 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 16:42:46.544013 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:46.536251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37911 (local address 127.0.98.3:36365)
0927 16:42:46.536873 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:46.536911 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:46.538127 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:46.538583 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:46.538590 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:46.538670 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:42:46.539134 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:46.539144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.539889 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.539891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:46.541161 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:46.541171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.541736 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.541740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:46.541896 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:46.542466 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:42:46.542468 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:42:46.542516 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:42:46.542590 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:42:46.542873 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:42:46.542882 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:42:46.542887 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 16:42:46.543012 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 16:42:46.543289 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":109,"threads_started":1}
I20260927 16:42:46.544981 20604 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37911
I20260927 16:42:46.545184 20604 heartbeater.cc:464] Registering TS with master...
I20260927 16:42:46.545631 20604 heartbeater.cc:511] Master 127.0.98.62:37911 requested a full tablet report, sending...
I20260927 16:42:46.546557 20154 ts_manager.cc:194] Registered new tserver with Master: c2a7ecb4cfd9401d97c4dc1ff862386b (127.0.98.3:39049)
I20260927 16:42:46.547070 20154 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:36365
I20260927 16:42:46.554608 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:42:46.560024 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:46.556468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38476 (local address 127.0.98.62:37911)
0927 16:42:46.556629 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:46.556634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:46.556803 (+ 169us) server_negotiation.cc:407] Connection header received
0927 16:42:46.556937 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:46.556940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:46.556988 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:46.557070 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:46.557348 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.557873 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.558667 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.558860 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.558959 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:46.559650 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:46.559654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:46.559656 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:42:46.559673 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:46.559688 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:42:46.559695 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:46.559765 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:46.559863 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 16:42:46.559908 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 16:42:46.562340 20154 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:42:46.565336 20154 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:42:46.579633 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:46.573520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39049 (local address 127.0.0.1:39510)
0927 16:42:46.573826 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:46.573875 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:46.573991 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:46.574910 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:46.574915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:46.574934 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:46.575054 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:46.575061 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.576356 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.576362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:46.577953 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:46.577963 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.579082 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.579085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:46.579184 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:46.579244 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 16:42:46.579343 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 16:42:46.579484 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":78,"thread_start_us":83,"threads_started":1}
I20260927 16:42:46.579638 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:46.573675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39510 (local address 127.0.98.3:39049)
0927 16:42:46.573955 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:46.573967 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:46.574001 (+ 34us) server_negotiation.cc:407] Connection header received
0927 16:42:46.574121 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:46.574128 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:46.574747 (+ 619us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:46.574926 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:46.575184 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.576245 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.578088 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.578956 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.579149 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:46.579334 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:46.579385 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 16:42:46.579488 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":91,"threads_started":1}
I20260927 16:42:46.579846 20619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:46.574509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45429 (local address 127.0.0.1:55690)
0927 16:42:46.574695 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:46.574707 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:46.574778 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:46.575814 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:46.575817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:46.575833 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:46.575928 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:46.575933 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.577197 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.577203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:46.578372 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:46.578381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.579483 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.579486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:46.579572 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:46.579593 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 16:42:46.579653 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 16:42:46.579720 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":22,"thread_start_us":74,"threads_started":1}
I20260927 16:42:46.579963 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:46.574880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55690 (local address 127.0.98.1:45429)
0927 16:42:46.575257 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:46.575268 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:46.575290 (+ 22us) server_negotiation.cc:407] Connection header received
0927 16:42:46.575356 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:46.575361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:46.575534 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:46.575685 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:46.576053 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.577093 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.578513 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.579346 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.579482 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:46.579651 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:46.579699 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 16:42:46.579786 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":148,"threads_started":1}
I20260927 16:42:46.580716 20617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:46.574182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37851 (local address 127.0.0.1:33128)
0927 16:42:46.574396 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:46.574410 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:46.574496 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:46.575436 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:46.575439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:46.575455 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:46.575544 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:46.575552 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.576853 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.576856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:46.577990 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:46.578001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.578969 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.578972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:46.580398 (+ 1426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:46.580440 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 16:42:46.580516 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 16:42:46.580582 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":47,"thread_start_us":100,"threads_started":1}
I20260927 16:42:46.580832 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:46.574352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33128 (local address 127.0.98.2:37851)
0927 16:42:46.574980 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:46.574994 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:46.575021 (+ 27us) server_negotiation.cc:407] Connection header received
0927 16:42:46.575098 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:46.575103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:46.575300 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:46.575456 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:46.575638 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.576691 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.578094 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.578826 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.578959 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:46.579146 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:46.580569 (+ 1423us) server_negotiation.cc:299] Negotiation successful
0927 16:42:46.580666 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":128,"threads_started":1}
I20260927 16:42:46.582382 20538 tablet_service.cc:1467] Processing CreateTablet for tablet ebec1d0173294a9a8d0d2a2ccbda70b1 (DEFAULT_TABLE table=test-table [id=b7d6d053f2ee4402acc507ec1a428c86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:42:46.582382 20276 tablet_service.cc:1467] Processing CreateTablet for tablet ebec1d0173294a9a8d0d2a2ccbda70b1 (DEFAULT_TABLE table=test-table [id=b7d6d053f2ee4402acc507ec1a428c86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:42:46.583302 20276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebec1d0173294a9a8d0d2a2ccbda70b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:46.583307 20538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebec1d0173294a9a8d0d2a2ccbda70b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:46.585700 20407 tablet_service.cc:1467] Processing CreateTablet for tablet ebec1d0173294a9a8d0d2a2ccbda70b1 (DEFAULT_TABLE table=test-table [id=b7d6d053f2ee4402acc507ec1a428c86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:42:46.586633 20407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebec1d0173294a9a8d0d2a2ccbda70b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:46.589944 20621 tablet_bootstrap.cc:492] T ebec1d0173294a9a8d0d2a2ccbda70b1 P c2a7ecb4cfd9401d97c4dc1ff862386b: Bootstrap starting.
I20260927 16:42:46.591740 20622 tablet_bootstrap.cc:492] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252: Bootstrap starting.
I20260927 16:42:46.592274 20621 tablet_bootstrap.cc:654] T ebec1d0173294a9a8d0d2a2ccbda70b1 P c2a7ecb4cfd9401d97c4dc1ff862386b: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:46.592888 20621 log.cc:824] T ebec1d0173294a9a8d0d2a2ccbda70b1 P c2a7ecb4cfd9401d97c4dc1ff862386b: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:46.593255 20623 tablet_bootstrap.cc:492] T ebec1d0173294a9a8d0d2a2ccbda70b1 P b62d589cfc2f45b58b7c464d09125841: Bootstrap starting.
I20260927 16:42:46.593669 20622 tablet_bootstrap.cc:654] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:46.594439 20622 log.cc:824] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:46.595052 20623 tablet_bootstrap.cc:654] T ebec1d0173294a9a8d0d2a2ccbda70b1 P b62d589cfc2f45b58b7c464d09125841: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:46.595261 20621 tablet_bootstrap.cc:492] T ebec1d0173294a9a8d0d2a2ccbda70b1 P c2a7ecb4cfd9401d97c4dc1ff862386b: No bootstrap required, opened a new log
I20260927 16:42:46.595505 20621 ts_tablet_manager.cc:1397] T ebec1d0173294a9a8d0d2a2ccbda70b1 P c2a7ecb4cfd9401d97c4dc1ff862386b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:42:46.595867 20623 log.cc:824] T ebec1d0173294a9a8d0d2a2ccbda70b1 P b62d589cfc2f45b58b7c464d09125841: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:46.596699 20622 tablet_bootstrap.cc:492] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252: No bootstrap required, opened a new log
I20260927 16:42:46.596875 20622 ts_tablet_manager.cc:1397] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 16:42:46.597599 20623 tablet_bootstrap.cc:492] T ebec1d0173294a9a8d0d2a2ccbda70b1 P b62d589cfc2f45b58b7c464d09125841: No bootstrap required, opened a new log
I20260927 16:42:46.597800 20623 ts_tablet_manager.cc:1397] T ebec1d0173294a9a8d0d2a2ccbda70b1 P b62d589cfc2f45b58b7c464d09125841: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 16:42:46.599819 20621 raft_consensus.cc:348] T ebec1d0173294a9a8d0d2a2ccbda70b1 P c2a7ecb4cfd9401d97c4dc1ff862386b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a7ecb4cfd9401d97c4dc1ff862386b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39049 } } peers { permanent_uuid: "b62d589cfc2f45b58b7c464d09125841" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37851 } } peers { permanent_uuid: "53ea075b7e23435fbb3b7fb81e0b0252" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45429 } }
I20260927 16:42:46.600097 20621 raft_consensus.cc:374] T ebec1d0173294a9a8d0d2a2ccbda70b1 P c2a7ecb4cfd9401d97c4dc1ff862386b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:46.600142 20621 raft_consensus.cc:729] T ebec1d0173294a9a8d0d2a2ccbda70b1 P c2a7ecb4cfd9401d97c4dc1ff862386b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a7ecb4cfd9401d97c4dc1ff862386b, State: Initialized, Role: FOLLOWER
I20260927 16:42:46.600591 20621 consensus_queue.cc:260] T ebec1d0173294a9a8d0d2a2ccbda70b1 P c2a7ecb4cfd9401d97c4dc1ff862386b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a7ecb4cfd9401d97c4dc1ff862386b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39049 } } peers { permanent_uuid: "b62d589cfc2f45b58b7c464d09125841" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37851 } } peers { permanent_uuid: "53ea075b7e23435fbb3b7fb81e0b0252" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45429 } }
I20260927 16:42:46.600881 20622 raft_consensus.cc:348] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a7ecb4cfd9401d97c4dc1ff862386b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39049 } } peers { permanent_uuid: "b62d589cfc2f45b58b7c464d09125841" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37851 } } peers { permanent_uuid: "53ea075b7e23435fbb3b7fb81e0b0252" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45429 } }
I20260927 16:42:46.601101 20622 raft_consensus.cc:374] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:46.601150 20622 raft_consensus.cc:729] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53ea075b7e23435fbb3b7fb81e0b0252, State: Initialized, Role: FOLLOWER
I20260927 16:42:46.601430 20604 heartbeater.cc:503] Master 127.0.98.62:37911 was elected leader, sending a full tablet report...
I20260927 16:42:46.601611 20621 ts_tablet_manager.cc:1428] T ebec1d0173294a9a8d0d2a2ccbda70b1 P c2a7ecb4cfd9401d97c4dc1ff862386b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:42:46.601611 20622 consensus_queue.cc:260] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a7ecb4cfd9401d97c4dc1ff862386b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39049 } } peers { permanent_uuid: "b62d589cfc2f45b58b7c464d09125841" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37851 } } peers { permanent_uuid: "53ea075b7e23435fbb3b7fb81e0b0252" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45429 } }
I20260927 16:42:46.602268 20342 heartbeater.cc:503] Master 127.0.98.62:37911 was elected leader, sending a full tablet report...
I20260927 16:42:46.602576 20622 ts_tablet_manager.cc:1428] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 16:42:46.602809 20623 raft_consensus.cc:348] T ebec1d0173294a9a8d0d2a2ccbda70b1 P b62d589cfc2f45b58b7c464d09125841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a7ecb4cfd9401d97c4dc1ff862386b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39049 } } peers { permanent_uuid: "b62d589cfc2f45b58b7c464d09125841" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37851 } } peers { permanent_uuid: "53ea075b7e23435fbb3b7fb81e0b0252" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45429 } }
I20260927 16:42:46.603021 20623 raft_consensus.cc:374] T ebec1d0173294a9a8d0d2a2ccbda70b1 P b62d589cfc2f45b58b7c464d09125841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:46.603080 20623 raft_consensus.cc:729] T ebec1d0173294a9a8d0d2a2ccbda70b1 P b62d589cfc2f45b58b7c464d09125841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b62d589cfc2f45b58b7c464d09125841, State: Initialized, Role: FOLLOWER
I20260927 16:42:46.603510 20623 consensus_queue.cc:260] T ebec1d0173294a9a8d0d2a2ccbda70b1 P b62d589cfc2f45b58b7c464d09125841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a7ecb4cfd9401d97c4dc1ff862386b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39049 } } peers { permanent_uuid: "b62d589cfc2f45b58b7c464d09125841" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37851 } } peers { permanent_uuid: "53ea075b7e23435fbb3b7fb81e0b0252" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45429 } }
I20260927 16:42:46.604096 20473 heartbeater.cc:503] Master 127.0.98.62:37911 was elected leader, sending a full tablet report...
I20260927 16:42:46.604364 20623 ts_tablet_manager.cc:1428] T ebec1d0173294a9a8d0d2a2ccbda70b1 P b62d589cfc2f45b58b7c464d09125841: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260927 16:42:46.651921 20343 tablet.cc:2369] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:42:46.785171 20605 tablet.cc:2369] T ebec1d0173294a9a8d0d2a2ccbda70b1 P c2a7ecb4cfd9401d97c4dc1ff862386b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:42:46.845674 20474 tablet.cc:2369] T ebec1d0173294a9a8d0d2a2ccbda70b1 P b62d589cfc2f45b58b7c464d09125841: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:42:46.894555 20628 raft_consensus.cc:482] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:42:46.894762 20628 raft_consensus.cc:504] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a7ecb4cfd9401d97c4dc1ff862386b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39049 } } peers { permanent_uuid: "b62d589cfc2f45b58b7c464d09125841" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37851 } } peers { permanent_uuid: "53ea075b7e23435fbb3b7fb81e0b0252" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45429 } }
I20260927 16:42:46.895916 20628 leader_election.cc:288] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a7ecb4cfd9401d97c4dc1ff862386b (127.0.98.3:39049), b62d589cfc2f45b58b7c464d09125841 (127.0.98.2:37851)
I20260927 16:42:46.900727 20630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:46.896173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39049 (local address 127.0.98.1:56269)
0927 16:42:46.896500 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:46.896515 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:46.896687 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:46.897271 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:46.897276 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:46.897308 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:46.897418 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:46.897423 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.898493 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.898496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:46.899493 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:46.899501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.900286 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.900290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:46.900385 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:46.900417 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 16:42:46.900513 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 16:42:46.900574 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":3,"thread_start_us":85,"threads_started":1}
I20260927 16:42:46.900738 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:46.896290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56269 (local address 127.0.98.3:39049)
0927 16:42:46.896451 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:46.896455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:46.896630 (+ 175us) server_negotiation.cc:407] Connection header received
0927 16:42:46.896998 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:46.897001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:46.897062 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:46.897111 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:46.897592 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.898352 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.899609 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.900167 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.900260 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:46.900318 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:46.900542 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 16:42:46.900614 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 16:42:46.901188 20631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:46.896574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37851 (local address 127.0.98.1:47281)
0927 16:42:46.896854 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:46.896867 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:42:46.896999 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:42:46.897237 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:42:46.897240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:42:46.897258 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:42:46.897411 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:46.897418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.898458 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.898461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:46.899741 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:42:46.899751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.900708 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.900712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:42:46.900807 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:46.900829 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 16:42:46.900981 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 16:42:46.901054 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":34,"spinlock_wait_cycles":12544,"thread_start_us":48,"threads_started":1}
I20260927 16:42:46.901222 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:46.896827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47281 (local address 127.0.98.2:37851)
0927 16:42:46.896975 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:46.896980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:46.896995 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:42:46.897081 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:46.897086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:46.897135 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:46.897207 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:42:46.897560 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.898338 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.899902 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.900503 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.900601 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:46.900650 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:46.901023 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 16:42:46.901113 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 16:42:46.901290 20558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebec1d0173294a9a8d0d2a2ccbda70b1" candidate_uuid: "53ea075b7e23435fbb3b7fb81e0b0252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a7ecb4cfd9401d97c4dc1ff862386b" is_pre_election: true
I20260927 16:42:46.901623 20558 raft_consensus.cc:2454] T ebec1d0173294a9a8d0d2a2ccbda70b1 P c2a7ecb4cfd9401d97c4dc1ff862386b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53ea075b7e23435fbb3b7fb81e0b0252 in term 0.
I20260927 16:42:46.901680 20427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebec1d0173294a9a8d0d2a2ccbda70b1" candidate_uuid: "53ea075b7e23435fbb3b7fb81e0b0252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b62d589cfc2f45b58b7c464d09125841" is_pre_election: true
I20260927 16:42:46.901903 20427 raft_consensus.cc:2454] T ebec1d0173294a9a8d0d2a2ccbda70b1 P b62d589cfc2f45b58b7c464d09125841 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53ea075b7e23435fbb3b7fb81e0b0252 in term 0.
I20260927 16:42:46.902137 20230 leader_election.cc:302] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53ea075b7e23435fbb3b7fb81e0b0252, c2a7ecb4cfd9401d97c4dc1ff862386b; no voters:
I20260927 16:42:46.902421 20628 raft_consensus.cc:2781] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:42:46.902513 20628 raft_consensus.cc:482] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:42:46.902555 20628 raft_consensus.cc:3037] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:46.903769 20628 raft_consensus.cc:504] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a7ecb4cfd9401d97c4dc1ff862386b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39049 } } peers { permanent_uuid: "b62d589cfc2f45b58b7c464d09125841" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37851 } } peers { permanent_uuid: "53ea075b7e23435fbb3b7fb81e0b0252" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45429 } }
I20260927 16:42:46.904078 20628 leader_election.cc:288] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252 [CANDIDATE]: Term 1 election: Requested vote from peers c2a7ecb4cfd9401d97c4dc1ff862386b (127.0.98.3:39049), b62d589cfc2f45b58b7c464d09125841 (127.0.98.2:37851)
I20260927 16:42:46.904397 20558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebec1d0173294a9a8d0d2a2ccbda70b1" candidate_uuid: "53ea075b7e23435fbb3b7fb81e0b0252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a7ecb4cfd9401d97c4dc1ff862386b"
I20260927 16:42:46.904517 20427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebec1d0173294a9a8d0d2a2ccbda70b1" candidate_uuid: "53ea075b7e23435fbb3b7fb81e0b0252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b62d589cfc2f45b58b7c464d09125841"
I20260927 16:42:46.904537 20558 raft_consensus.cc:3037] T ebec1d0173294a9a8d0d2a2ccbda70b1 P c2a7ecb4cfd9401d97c4dc1ff862386b [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:46.904632 20427 raft_consensus.cc:3037] T ebec1d0173294a9a8d0d2a2ccbda70b1 P b62d589cfc2f45b58b7c464d09125841 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:46.905509 20558 raft_consensus.cc:2454] T ebec1d0173294a9a8d0d2a2ccbda70b1 P c2a7ecb4cfd9401d97c4dc1ff862386b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53ea075b7e23435fbb3b7fb81e0b0252 in term 1.
I20260927 16:42:46.905773 20427 raft_consensus.cc:2454] T ebec1d0173294a9a8d0d2a2ccbda70b1 P b62d589cfc2f45b58b7c464d09125841 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53ea075b7e23435fbb3b7fb81e0b0252 in term 1.
I20260927 16:42:46.905797 20230 leader_election.cc:302] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53ea075b7e23435fbb3b7fb81e0b0252, c2a7ecb4cfd9401d97c4dc1ff862386b; no voters:
I20260927 16:42:46.905959 20628 raft_consensus.cc:2781] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:42:46.906199 20628 raft_consensus.cc:686] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252 [term 1 LEADER]: Becoming Leader. State: Replica: 53ea075b7e23435fbb3b7fb81e0b0252, State: Running, Role: LEADER
I20260927 16:42:46.906527 20628 consensus_queue.cc:237] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a7ecb4cfd9401d97c4dc1ff862386b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39049 } } peers { permanent_uuid: "b62d589cfc2f45b58b7c464d09125841" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37851 } } peers { permanent_uuid: "53ea075b7e23435fbb3b7fb81e0b0252" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45429 } }
I20260927 16:42:46.909731 20151 catalog_manager.cc:5495] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53ea075b7e23435fbb3b7fb81e0b0252 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "53ea075b7e23435fbb3b7fb81e0b0252" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2a7ecb4cfd9401d97c4dc1ff862386b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b62d589cfc2f45b58b7c464d09125841" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53ea075b7e23435fbb3b7fb81e0b0252" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45429 } health_report { overall_health: HEALTHY } } }
I20260927 16:42:46.951828 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:46.948197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55704 (local address 127.0.98.1:45429)
0927 16:42:46.948342 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:46.948346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:46.948363 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:42:46.948466 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:46.948468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:46.948523 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:46.948642 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:42:46.948882 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.949374 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.950280 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:46.950503 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:46.950607 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:46.950911 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:42:46.951431 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:42:46.951565 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:46.951626 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 16:42:46.951677 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 16:42:46.957897 20427 raft_consensus.cc:1261] T ebec1d0173294a9a8d0d2a2ccbda70b1 P b62d589cfc2f45b58b7c464d09125841 [term 1 FOLLOWER]: Refusing update from remote peer 53ea075b7e23435fbb3b7fb81e0b0252: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:42:46.957917 20558 raft_consensus.cc:1261] T ebec1d0173294a9a8d0d2a2ccbda70b1 P c2a7ecb4cfd9401d97c4dc1ff862386b [term 1 FOLLOWER]: Refusing update from remote peer 53ea075b7e23435fbb3b7fb81e0b0252: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:42:46.958648 20632 consensus_queue.cc:1035] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a7ecb4cfd9401d97c4dc1ff862386b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:42:46.958822 20628 consensus_queue.cc:1035] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b62d589cfc2f45b58b7c464d09125841" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:42:46.967271 20637 mvcc.cc:205] Tried to move back new op lower bound from 7334000095052410880 to 7334000094852177920. Current Snapshot: MvccSnapshot[applied={T|T < 7334000095052410880}]
I20260927 16:42:46.967653 20636 mvcc.cc:205] Tried to move back new op lower bound from 7334000095052410880 to 7334000094852177920. Current Snapshot: MvccSnapshot[applied={T|T < 7334000095052410880}]
I20260927 16:42:46.974978 20151 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:42:46.975165 20151 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:42:46.977480 20151 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f3a4966e21f4c22a40d8dc21dd18f53: Sending DeleteTablet for 3 replicas of tablet ebec1d0173294a9a8d0d2a2ccbda70b1
I20260927 16:42:46.978317 20538 tablet_service.cc:1514] Processing DeleteTablet for tablet ebec1d0173294a9a8d0d2a2ccbda70b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:42:46 UTC) from {username='slave'} at 127.0.0.1:39510
I20260927 16:42:46.978387 20407 tablet_service.cc:1514] Processing DeleteTablet for tablet ebec1d0173294a9a8d0d2a2ccbda70b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:42:46 UTC) from {username='slave'} at 127.0.0.1:33128
I20260927 16:42:46.978379 20276 tablet_service.cc:1514] Processing DeleteTablet for tablet ebec1d0173294a9a8d0d2a2ccbda70b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:42:46 UTC) from {username='slave'} at 127.0.0.1:55690
I20260927 16:42:46.978893 20645 tablet_replica.cc:330] T ebec1d0173294a9a8d0d2a2ccbda70b1 P c2a7ecb4cfd9401d97c4dc1ff862386b: stopping tablet replica
I20260927 16:42:46.978960 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 20214
I20260927 16:42:46.978982 20646 tablet_replica.cc:330] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252: stopping tablet replica
I20260927 16:42:46.979195 20645 raft_consensus.cc:2229] T ebec1d0173294a9a8d0d2a2ccbda70b1 P c2a7ecb4cfd9401d97c4dc1ff862386b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:42:46.979271 20646 raft_consensus.cc:2229] T ebec1d0173294a9a8d0d2a2ccbda70b1 P 53ea075b7e23435fbb3b7fb81e0b0252 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:42:46.979539 20645 raft_consensus.cc:2258] T ebec1d0173294a9a8d0d2a2ccbda70b1 P c2a7ecb4cfd9401d97c4dc1ff862386b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:42:46.979925 20647 tablet_replica.cc:330] T ebec1d0173294a9a8d0d2a2ccbda70b1 P b62d589cfc2f45b58b7c464d09125841: stopping tablet replica
I20260927 16:42:46.980166 20647 raft_consensus.cc:2229] T ebec1d0173294a9a8d0d2a2ccbda70b1 P b62d589cfc2f45b58b7c464d09125841 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:42:46.980461 20647 raft_consensus.cc:2258] T ebec1d0173294a9a8d0d2a2ccbda70b1 P b62d589cfc2f45b58b7c464d09125841 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:42:46.981259 20645 ts_tablet_manager.cc:1905] T ebec1d0173294a9a8d0d2a2ccbda70b1 P c2a7ecb4cfd9401d97c4dc1ff862386b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:42:46.981949 20647 ts_tablet_manager.cc:1905] T ebec1d0173294a9a8d0d2a2ccbda70b1 P b62d589cfc2f45b58b7c464d09125841: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:42:46.984309 20647 ts_tablet_manager.cc:1918] T ebec1d0173294a9a8d0d2a2ccbda70b1 P b62d589cfc2f45b58b7c464d09125841: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:42:46.984442 20647 log.cc:1196] T ebec1d0173294a9a8d0d2a2ccbda70b1 P b62d589cfc2f45b58b7c464d09125841: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-1/wal/wals/ebec1d0173294a9a8d0d2a2ccbda70b1
I20260927 16:42:46.984615 20645 ts_tablet_manager.cc:1918] T ebec1d0173294a9a8d0d2a2ccbda70b1 P c2a7ecb4cfd9401d97c4dc1ff862386b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:42:46.984722 20645 log.cc:1196] T ebec1d0173294a9a8d0d2a2ccbda70b1 P c2a7ecb4cfd9401d97c4dc1ff862386b: Deleting WAL directory at /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790527121515808-392-0/minicluster-data/ts-2/wal/wals/ebec1d0173294a9a8d0d2a2ccbda70b1
I20260927 16:42:46.984745 20647 ts_tablet_manager.cc:1939] T ebec1d0173294a9a8d0d2a2ccbda70b1 P b62d589cfc2f45b58b7c464d09125841: Deleting consensus metadata
I20260927 16:42:46.985019 20645 ts_tablet_manager.cc:1939] T ebec1d0173294a9a8d0d2a2ccbda70b1 P c2a7ecb4cfd9401d97c4dc1ff862386b: Deleting consensus metadata
I20260927 16:42:46.985807 20139 catalog_manager.cc:4841] TS b62d589cfc2f45b58b7c464d09125841 (127.0.98.2:37851): tablet ebec1d0173294a9a8d0d2a2ccbda70b1 (table test-table [id=b7d6d053f2ee4402acc507ec1a428c86]) successfully deleted
I20260927 16:42:46.986153 20139 catalog_manager.cc:4841] TS c2a7ecb4cfd9401d97c4dc1ff862386b (127.0.98.3:39049): tablet ebec1d0173294a9a8d0d2a2ccbda70b1 (table test-table [id=b7d6d053f2ee4402acc507ec1a428c86]) successfully deleted
W20260927 16:42:46.986737 20139 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45429 (error 108)
I20260927 16:42:46.986886 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 20345
I20260927 16:42:46.992034 20617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:46.991784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45429 (local address 127.0.0.1:55712)
0927 16:42:46.991905 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 16:42:46.991968 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 16:42:46.992182 20139 catalog_manager.cc:4584] TS 53ea075b7e23435fbb3b7fb81e0b0252 (127.0.98.1:45429): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ebec1d0173294a9a8d0d2a2ccbda70b1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45429: connect: Connection refused (error 111)
I20260927 16:42:46.993145 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 20476
I20260927 16:42:46.998875 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 20123
2026-09-27T16:42:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1294 ms)
[----------] 6 tests from EncryptionPolicyTest (10266 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:42:47 dist-test-slave-z40b krb5kdc[20651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:42:47 dist-test-slave-z40b krb5kdc[20651](info): set up 2 sockets
Sep 27 16:42:47 dist-test-slave-z40b krb5kdc[20651](info): commencing operation
krb5kdc: starting...
W20260927 16:42:47.999320 392 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:42:48 dist-test-slave-z40b krb5kdc[20651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:42:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:42:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:42:48.112155 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35711
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:42:48.216481 20664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:48.216711 20664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:48.216750 20664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:48.220553 20664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:42:48.220638 20664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:48.220679 20664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:48.220695 20664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:42:48.220710 20664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:42:48.225577 20664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35711
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:48.226437 20664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:48.227409 20664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:48.235311 20670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:48.235324 20669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:48.235380 20664 server_base.cc:1034] running on GCE node
W20260927 16:42:48.235316 20672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:48.236018 20664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:48.237347 20664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:48.238571 20664 hybrid_clock.cc:648] HybridClock initialized: now 1790527368238555 us; error 37 us; skew 500 ppm
Sep 27 16:42:48 dist-test-slave-z40b krb5kdc[20651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:48.241173 20664 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:42:48.242041 20664 webserver.cc:460] Webserver started at http://127.0.0.1:38571/ using document root <none> and password file <none>
I20260927 16:42:48.242561 20664 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:48.242619 20664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:48.242853 20664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:48.244881 20664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "bff5e4a6b942426abd04b47cd3e11b50"
format_stamp: "Formatted at 2026-09-27 16:42:48 on dist-test-slave-z40b"
server_key: "ea35192210929b833d04f89b0777ee0a"
server_key_iv: "217a74160871480c10744e8142348061"
server_key_version: "encryptionkey@0"
I20260927 16:42:48.245450 20664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "bff5e4a6b942426abd04b47cd3e11b50"
format_stamp: "Formatted at 2026-09-27 16:42:48 on dist-test-slave-z40b"
server_key: "ea35192210929b833d04f89b0777ee0a"
server_key_iv: "217a74160871480c10744e8142348061"
server_key_version: "encryptionkey@0"
I20260927 16:42:48.248916 20664 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:42:48.251058 20679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:48.252233 20664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:42:48.252367 20664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "bff5e4a6b942426abd04b47cd3e11b50"
format_stamp: "Formatted at 2026-09-27 16:42:48 on dist-test-slave-z40b"
server_key: "ea35192210929b833d04f89b0777ee0a"
server_key_iv: "217a74160871480c10744e8142348061"
server_key_version: "encryptionkey@0"
I20260927 16:42:48.252496 20664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:48.265825 20664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:48.268415 20664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:48.268601 20664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:48.275035 20664 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35711
I20260927 16:42:48.275048 20731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35711 every 8 connection(s)
I20260927 16:42:48.275933 20664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:42:48.277063 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 20664
I20260927 16:42:48.277199 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:42:48.277505 392 external_mini_cluster.cc:1428] Setting key c01f33083ab8b1a9172ed2b12d5dc420
I20260927 16:42:48.278942 20732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:48.285060 20732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bff5e4a6b942426abd04b47cd3e11b50: Bootstrap starting.
Sep 27 16:42:48 dist-test-slave-z40b krb5kdc[20651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 16:42:48.287304 20732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bff5e4a6b942426abd04b47cd3e11b50: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:48.288098 20732 log.cc:824] T 00000000000000000000000000000000 P bff5e4a6b942426abd04b47cd3e11b50: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:48.289845 20732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bff5e4a6b942426abd04b47cd3e11b50: No bootstrap required, opened a new log
I20260927 16:42:48.291149 20735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:48.278719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36807 (local address 127.0.0.1:35711)
0927 16:42:48.279071 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:48.279089 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:48.279119 (+ 30us) server_negotiation.cc:407] Connection header received
0927 16:42:48.279827 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:48.279862 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:48.280171 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:48.280440 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:48.281306 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:48.282223 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:48.283403 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:48.283844 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:48.283948 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:48.286562 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:48.286566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:48.286577 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:48.286604 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:48.288399 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:48.288917 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:48.288922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:48.288926 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:48.289026 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:48.289380 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:48.289390 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:48.289392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:48.289788 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:48.289995 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:48.290302 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 16:42:48.290511 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":138,"threads_started":1}
I20260927 16:42:48.293672 20732 raft_consensus.cc:348] T 00000000000000000000000000000000 P bff5e4a6b942426abd04b47cd3e11b50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff5e4a6b942426abd04b47cd3e11b50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35711 } }
I20260927 16:42:48.293943 20732 raft_consensus.cc:374] T 00000000000000000000000000000000 P bff5e4a6b942426abd04b47cd3e11b50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:48.293995 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:42:48.294000 20732 raft_consensus.cc:729] T 00000000000000000000000000000000 P bff5e4a6b942426abd04b47cd3e11b50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bff5e4a6b942426abd04b47cd3e11b50, State: Initialized, Role: FOLLOWER
I20260927 16:42:48.294490 20732 consensus_queue.cc:260] T 00000000000000000000000000000000 P bff5e4a6b942426abd04b47cd3e11b50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff5e4a6b942426abd04b47cd3e11b50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35711 } }
I20260927 16:42:48.294631 20732 raft_consensus.cc:388] T 00000000000000000000000000000000 P bff5e4a6b942426abd04b47cd3e11b50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:42:48.294689 20732 raft_consensus.cc:482] T 00000000000000000000000000000000 P bff5e4a6b942426abd04b47cd3e11b50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:42:48.294771 20732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bff5e4a6b942426abd04b47cd3e11b50 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:48.295704 20732 raft_consensus.cc:504] T 00000000000000000000000000000000 P bff5e4a6b942426abd04b47cd3e11b50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff5e4a6b942426abd04b47cd3e11b50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35711 } }
I20260927 16:42:48.296070 20732 leader_election.cc:302] T 00000000000000000000000000000000 P bff5e4a6b942426abd04b47cd3e11b50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bff5e4a6b942426abd04b47cd3e11b50; no voters:
I20260927 16:42:48.296370 20732 leader_election.cc:288] T 00000000000000000000000000000000 P bff5e4a6b942426abd04b47cd3e11b50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:42:48.296518 20742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bff5e4a6b942426abd04b47cd3e11b50 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:42:48.296790 20742 raft_consensus.cc:686] T 00000000000000000000000000000000 P bff5e4a6b942426abd04b47cd3e11b50 [term 1 LEADER]: Becoming Leader. State: Replica: bff5e4a6b942426abd04b47cd3e11b50, State: Running, Role: LEADER
I20260927 16:42:48.297153 20742 consensus_queue.cc:237] T 00000000000000000000000000000000 P bff5e4a6b942426abd04b47cd3e11b50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff5e4a6b942426abd04b47cd3e11b50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35711 } }
I20260927 16:42:48.297518 20732 sys_catalog.cc:564] T 00000000000000000000000000000000 P bff5e4a6b942426abd04b47cd3e11b50 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:42:48.298331 20742 sys_catalog.cc:455] T 00000000000000000000000000000000 P bff5e4a6b942426abd04b47cd3e11b50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bff5e4a6b942426abd04b47cd3e11b50. Latest consensus state: current_term: 1 leader_uuid: "bff5e4a6b942426abd04b47cd3e11b50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff5e4a6b942426abd04b47cd3e11b50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35711 } } }
I20260927 16:42:48.298439 20742 sys_catalog.cc:458] T 00000000000000000000000000000000 P bff5e4a6b942426abd04b47cd3e11b50 [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:48.298700 20744 sys_catalog.cc:455] T 00000000000000000000000000000000 P bff5e4a6b942426abd04b47cd3e11b50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bff5e4a6b942426abd04b47cd3e11b50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff5e4a6b942426abd04b47cd3e11b50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35711 } } }
I20260927 16:42:48.298785 20744 sys_catalog.cc:458] T 00000000000000000000000000000000 P bff5e4a6b942426abd04b47cd3e11b50 [sys.catalog]: This master's current role is: LEADER
W20260927 16:42:48.302048 20757 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bff5e4a6b942426abd04b47cd3e11b50: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:42:48.302126 20757 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:42:48.302219 20751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:42:48.302863 20751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:42:48.304507 20735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:48.296019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46003 (local address 127.0.0.1:35711)
0927 16:42:48.296139 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:48.296143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:48.296266 (+ 123us) server_negotiation.cc:407] Connection header received
0927 16:42:48.296350 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:48.296352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:48.296396 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:48.296484 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:48.297323 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:48.298061 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:48.299111 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:48.299459 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:48.299534 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:48.301134 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:48.301138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:48.301141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:48.301163 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:48.302699 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:48.303187 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:48.303190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:48.303192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:48.303291 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:48.303632 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:48.303637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:48.303638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:48.303821 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:48.303970 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:48.304264 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 16:42:48.304353 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:42:48.307742 20751 catalog_manager.cc:1347] Generated new cluster ID: 4f47e2b804344031a9e7bad3a9284ef4
I20260927 16:42:48.307801 20751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:42:48.321484 20751 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:42:48.322286 20751 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:42:48.332477 20751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bff5e4a6b942426abd04b47cd3e11b50: Generated new TSK 0
I20260927 16:42:48.333125 20751 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:42:48.337702 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 20664
W20260927 16:42:48.343356 20649 connection.cc:535] client connection to 127.0.0.1:35711 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35711): Connection reset by peer (error 104)
2026-09-27T16:42:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:42:48 dist-test-slave-z40b krb5kdc[20763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:42:48 dist-test-slave-z40b krb5kdc[20763](info): set up 2 sockets
Sep 27 16:42:48 dist-test-slave-z40b krb5kdc[20763](info): commencing operation
krb5kdc: starting...
W20260927 16:42:49.355074 392 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:42:49 dist-test-slave-z40b krb5kdc[20763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:42:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:42:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:42:49.469540 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40379
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:42:49.573541 20776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:49.573815 20776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:49.573884 20776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:49.577823 20776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:42:49.577901 20776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:49.577965 20776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:49.578001 20776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:42:49.578017 20776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:42:49.582988 20776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40379
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:49.583948 20776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:49.584825 20776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:49.592850 20782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:49.592865 20784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:49.592860 20781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:49.593212 20776 server_base.cc:1034] running on GCE node
I20260927 16:42:49.593753 20776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:49.594745 20776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:49.595923 20776 hybrid_clock.cc:648] HybridClock initialized: now 1790527369595916 us; error 25 us; skew 500 ppm
Sep 27 16:42:49 dist-test-slave-z40b krb5kdc[20763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:49.599676 20776 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:42:49.600498 20776 webserver.cc:460] Webserver started at http://127.0.0.1:40693/ using document root <none> and password file <none>
I20260927 16:42:49.601037 20776 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:49.601122 20776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:49.601317 20776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:49.603343 20776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "181c621569984d9c9df1116c8596b07f"
format_stamp: "Formatted at 2026-09-27 16:42:49 on dist-test-slave-z40b"
server_key: "5739d8edfdc169304432973feabf8321"
server_key_iv: "f32b8d235ada112179bc94a99cf6eed9"
server_key_version: "encryptionkey@0"
I20260927 16:42:49.603896 20776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "181c621569984d9c9df1116c8596b07f"
format_stamp: "Formatted at 2026-09-27 16:42:49 on dist-test-slave-z40b"
server_key: "5739d8edfdc169304432973feabf8321"
server_key_iv: "f32b8d235ada112179bc94a99cf6eed9"
server_key_version: "encryptionkey@0"
I20260927 16:42:49.607412 20776 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:42:49.609630 20791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:49.610780 20776 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:42:49.610915 20776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "181c621569984d9c9df1116c8596b07f"
format_stamp: "Formatted at 2026-09-27 16:42:49 on dist-test-slave-z40b"
server_key: "5739d8edfdc169304432973feabf8321"
server_key_iv: "f32b8d235ada112179bc94a99cf6eed9"
server_key_version: "encryptionkey@0"
I20260927 16:42:49.611025 20776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:49.632755 20776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:49.635473 20776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:49.635663 20776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:49.642324 20776 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40379
I20260927 16:42:49.642382 20843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40379 every 8 connection(s)
I20260927 16:42:49.643256 20776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:42:49.644719 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 20776
I20260927 16:42:49.644865 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:42:49.645174 392 external_mini_cluster.cc:1428] Setting key 7d13f2c7d7eb431a6e18bd15c095a90b
I20260927 16:42:49.646347 20844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:49.653041 20844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 181c621569984d9c9df1116c8596b07f: Bootstrap starting.
I20260927 16:42:49.655035 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:49.646460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45551 (local address 127.0.0.1:40379)
0927 16:42:49.646874 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:49.646892 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:49.646933 (+ 41us) server_negotiation.cc:407] Connection header received
0927 16:42:49.647777 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:49.647799 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:49.648213 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:49.648558 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:49.648939 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:49.650365 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:49.651364 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:49.652099 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:49.652249 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:49.653125 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:49.653131 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:49.653147 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:42:49.653650 (+ 503us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:49.653711 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:42:49.653740 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:49.653863 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:49.654011 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 16:42:49.654142 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":101,"threads_started":1}
I20260927 16:42:49.655680 20844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 181c621569984d9c9df1116c8596b07f: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:49.656425 20844 log.cc:824] T 00000000000000000000000000000000 P 181c621569984d9c9df1116c8596b07f: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:49.658056 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:42:49.658608 20844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 181c621569984d9c9df1116c8596b07f: No bootstrap required, opened a new log
I20260927 16:42:49.663017 20844 raft_consensus.cc:348] T 00000000000000000000000000000000 P 181c621569984d9c9df1116c8596b07f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181c621569984d9c9df1116c8596b07f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40379 } }
I20260927 16:42:49.663326 20844 raft_consensus.cc:374] T 00000000000000000000000000000000 P 181c621569984d9c9df1116c8596b07f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:49.663443 20844 raft_consensus.cc:729] T 00000000000000000000000000000000 P 181c621569984d9c9df1116c8596b07f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 181c621569984d9c9df1116c8596b07f, State: Initialized, Role: FOLLOWER
I20260927 16:42:49.663908 20844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 181c621569984d9c9df1116c8596b07f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181c621569984d9c9df1116c8596b07f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40379 } }
I20260927 16:42:49.664080 20844 raft_consensus.cc:388] T 00000000000000000000000000000000 P 181c621569984d9c9df1116c8596b07f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:42:49.664172 20844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 181c621569984d9c9df1116c8596b07f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:42:49.664265 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:49.660180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54061 (local address 127.0.0.1:40379)
0927 16:42:49.660296 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:49.660300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:49.660481 (+ 181us) server_negotiation.cc:407] Connection header received
0927 16:42:49.660587 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:49.660588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:49.660620 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:49.660679 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:49.660986 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:49.661447 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:49.662540 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:49.662751 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:49.662820 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:49.663783 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:49.663786 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:49.663789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:42:49.663799 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:49.663822 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:42:49.663833 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:49.663908 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:49.664035 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 16:42:49.664067 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:42:49.664291 20844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 181c621569984d9c9df1116c8596b07f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:49.665247 20844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 181c621569984d9c9df1116c8596b07f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181c621569984d9c9df1116c8596b07f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40379 } }
I20260927 16:42:49.665519 20844 leader_election.cc:302] T 00000000000000000000000000000000 P 181c621569984d9c9df1116c8596b07f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 181c621569984d9c9df1116c8596b07f; no voters:
I20260927 16:42:49.665737 20844 leader_election.cc:288] T 00000000000000000000000000000000 P 181c621569984d9c9df1116c8596b07f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:42:49.665845 20854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 181c621569984d9c9df1116c8596b07f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:42:49.666097 20854 raft_consensus.cc:686] T 00000000000000000000000000000000 P 181c621569984d9c9df1116c8596b07f [term 1 LEADER]: Becoming Leader. State: Replica: 181c621569984d9c9df1116c8596b07f, State: Running, Role: LEADER
I20260927 16:42:49.666399 20854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 181c621569984d9c9df1116c8596b07f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181c621569984d9c9df1116c8596b07f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40379 } }
I20260927 16:42:49.666893 20844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 181c621569984d9c9df1116c8596b07f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:42:49.668040 20855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181c621569984d9c9df1116c8596b07f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "181c621569984d9c9df1116c8596b07f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181c621569984d9c9df1116c8596b07f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40379 } } }
I20260927 16:42:49.668170 20855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181c621569984d9c9df1116c8596b07f [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:49.668478 20856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181c621569984d9c9df1116c8596b07f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 181c621569984d9c9df1116c8596b07f. Latest consensus state: current_term: 1 leader_uuid: "181c621569984d9c9df1116c8596b07f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181c621569984d9c9df1116c8596b07f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40379 } } }
I20260927 16:42:49.668635 20856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181c621569984d9c9df1116c8596b07f [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:49.668562 20863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:42:49.671865 20863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:42:49.676671 20863 catalog_manager.cc:1347] Generated new cluster ID: c9709862e9ef4e61800fc2a4f4e88b79
I20260927 16:42:49.676743 20863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:42:49.690685 20863 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:42:49.691581 20863 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:42:49.702697 20863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 181c621569984d9c9df1116c8596b07f: Generated new TSK 0
I20260927 16:42:49.703480 20863 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:42:49.707728 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 20776
W20260927 16:42:49.712965 20761 connection.cc:535] client connection to 127.0.0.1:40379 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40379): Connection reset by peer (error 104)
2026-09-27T16:42:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:42:49 dist-test-slave-z40b krb5kdc[20875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:42:49 dist-test-slave-z40b krb5kdc[20875](info): set up 2 sockets
Sep 27 16:42:49 dist-test-slave-z40b krb5kdc[20875](info): commencing operation
krb5kdc: starting...
W20260927 16:42:50.715507 392 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:42:50 dist-test-slave-z40b krb5kdc[20875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:42:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:42:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:42:50.827010 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35183
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:42:50.931722 20888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:50.931943 20888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:50.931982 20888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:50.935729 20888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:42:50.935773 20888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:50.935791 20888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:50.935811 20888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:42:50.935855 20888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:42:50.940660 20888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35183
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:50.941496 20888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:50.942396 20888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:50.950074 20893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:50.950133 20894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:50.950120 20888 server_base.cc:1034] running on GCE node
W20260927 16:42:50.950261 20896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:50.950759 20888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:50.951747 20888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:50.952922 20888 hybrid_clock.cc:648] HybridClock initialized: now 1790527370952905 us; error 27 us; skew 500 ppm
Sep 27 16:42:50 dist-test-slave-z40b krb5kdc[20875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:50.955598 20888 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:42:50.956331 20888 webserver.cc:460] Webserver started at http://127.0.0.1:33811/ using document root <none> and password file <none>
I20260927 16:42:50.956847 20888 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:50.956934 20888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:50.957149 20888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:50.959002 20888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "977c4a005b424955baa0676f9be5ba05"
format_stamp: "Formatted at 2026-09-27 16:42:50 on dist-test-slave-z40b"
server_key: "ecdac6cc836048d343ed673ce47c0748"
server_key_iv: "7d33aa8fb088b067c9fe587e918dfe4d"
server_key_version: "encryptionkey@0"
I20260927 16:42:50.959543 20888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "977c4a005b424955baa0676f9be5ba05"
format_stamp: "Formatted at 2026-09-27 16:42:50 on dist-test-slave-z40b"
server_key: "ecdac6cc836048d343ed673ce47c0748"
server_key_iv: "7d33aa8fb088b067c9fe587e918dfe4d"
server_key_version: "encryptionkey@0"
I20260927 16:42:50.962895 20888 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:42:50.965014 20903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:50.966238 20888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:42:50.966367 20888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "977c4a005b424955baa0676f9be5ba05"
format_stamp: "Formatted at 2026-09-27 16:42:50 on dist-test-slave-z40b"
server_key: "ecdac6cc836048d343ed673ce47c0748"
server_key_iv: "7d33aa8fb088b067c9fe587e918dfe4d"
server_key_version: "encryptionkey@0"
I20260927 16:42:50.966478 20888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:50.970608 20888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:50.973160 20888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:50.973330 20888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:50.979185 20888 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35183
I20260927 16:42:50.979280 20955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35183 every 8 connection(s)
I20260927 16:42:50.980024 20888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:42:50.982398 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 20888
I20260927 16:42:50.982419 20956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:50.982523 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:42:50.982784 392 external_mini_cluster.cc:1428] Setting key c6f0ece6a94a62f969c74d16ce562d62
I20260927 16:42:50.988313 20959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:50.983981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60069 (local address 127.0.0.1:35183)
0927 16:42:50.984376 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:50.984394 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:50.984433 (+ 39us) server_negotiation.cc:407] Connection header received
0927 16:42:50.985564 (+ 1131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:50.985599 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:50.985956 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:50.986312 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:50.986996 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:50.987001 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:50.987015 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:42:50.987026 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:50.987083 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:42:50.987107 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:50.987270 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:50.987362 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 16:42:50.987499 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":109,"threads_started":1}
I20260927 16:42:50.989238 20956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 977c4a005b424955baa0676f9be5ba05: Bootstrap starting.
I20260927 16:42:50.990952 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:42:50.991725 20956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 977c4a005b424955baa0676f9be5ba05: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:50.992453 20956 log.cc:824] T 00000000000000000000000000000000 P 977c4a005b424955baa0676f9be5ba05: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:50.994591 20959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:50.993058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38437 (local address 127.0.0.1:35183)
0927 16:42:50.993185 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:50.993190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:50.993428 (+ 238us) server_negotiation.cc:407] Connection header received
0927 16:42:50.993530 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:50.993532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:50.993567 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:50.993627 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:50.994223 (+ 596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:50.994225 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:50.994227 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:42:50.994236 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:50.994255 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:42:50.994264 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:50.994325 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:50.994435 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 16:42:50.994462 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 16:42:50.994683 20956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 977c4a005b424955baa0676f9be5ba05: No bootstrap required, opened a new log
I20260927 16:42:50.997962 20956 raft_consensus.cc:348] T 00000000000000000000000000000000 P 977c4a005b424955baa0676f9be5ba05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977c4a005b424955baa0676f9be5ba05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35183 } }
I20260927 16:42:50.998168 20956 raft_consensus.cc:374] T 00000000000000000000000000000000 P 977c4a005b424955baa0676f9be5ba05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:50.998198 20956 raft_consensus.cc:729] T 00000000000000000000000000000000 P 977c4a005b424955baa0676f9be5ba05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 977c4a005b424955baa0676f9be5ba05, State: Initialized, Role: FOLLOWER
I20260927 16:42:50.998564 20956 consensus_queue.cc:260] T 00000000000000000000000000000000 P 977c4a005b424955baa0676f9be5ba05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977c4a005b424955baa0676f9be5ba05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35183 } }
I20260927 16:42:50.998670 20956 raft_consensus.cc:388] T 00000000000000000000000000000000 P 977c4a005b424955baa0676f9be5ba05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:42:50.998711 20956 raft_consensus.cc:482] T 00000000000000000000000000000000 P 977c4a005b424955baa0676f9be5ba05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:42:50.998771 20956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 977c4a005b424955baa0676f9be5ba05 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:50.999576 20956 raft_consensus.cc:504] T 00000000000000000000000000000000 P 977c4a005b424955baa0676f9be5ba05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977c4a005b424955baa0676f9be5ba05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35183 } }
I20260927 16:42:50.999833 20956 leader_election.cc:302] T 00000000000000000000000000000000 P 977c4a005b424955baa0676f9be5ba05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 977c4a005b424955baa0676f9be5ba05; no voters:
I20260927 16:42:51.000046 20956 leader_election.cc:288] T 00000000000000000000000000000000 P 977c4a005b424955baa0676f9be5ba05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:42:51.000167 20966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 977c4a005b424955baa0676f9be5ba05 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:42:51.000391 20966 raft_consensus.cc:686] T 00000000000000000000000000000000 P 977c4a005b424955baa0676f9be5ba05 [term 1 LEADER]: Becoming Leader. State: Replica: 977c4a005b424955baa0676f9be5ba05, State: Running, Role: LEADER
I20260927 16:42:51.000703 20966 consensus_queue.cc:237] T 00000000000000000000000000000000 P 977c4a005b424955baa0676f9be5ba05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977c4a005b424955baa0676f9be5ba05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35183 } }
I20260927 16:42:51.001362 20956 sys_catalog.cc:564] T 00000000000000000000000000000000 P 977c4a005b424955baa0676f9be5ba05 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:42:51.002373 20967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 977c4a005b424955baa0676f9be5ba05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 977c4a005b424955baa0676f9be5ba05. Latest consensus state: current_term: 1 leader_uuid: "977c4a005b424955baa0676f9be5ba05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977c4a005b424955baa0676f9be5ba05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35183 } } }
I20260927 16:42:51.002491 20967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 977c4a005b424955baa0676f9be5ba05 [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:51.002771 20968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 977c4a005b424955baa0676f9be5ba05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "977c4a005b424955baa0676f9be5ba05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977c4a005b424955baa0676f9be5ba05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35183 } } }
I20260927 16:42:51.002858 20968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 977c4a005b424955baa0676f9be5ba05 [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:51.003294 20975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:42:51.006175 20975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:42:51.011584 20975 catalog_manager.cc:1347] Generated new cluster ID: 374d8dea1179472185b051761b62f505
I20260927 16:42:51.011643 20975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:42:51.034688 20975 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:42:51.034970 20975 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:42:51.043661 20975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 977c4a005b424955baa0676f9be5ba05: Generated new TSK 0
I20260927 16:42:51.044256 20975 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:42:51.051574 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 20888
2026-09-27T16:42:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:42:51 dist-test-slave-z40b krb5kdc[20987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:42:51 dist-test-slave-z40b krb5kdc[20987](info): set up 2 sockets
Sep 27 16:42:51 dist-test-slave-z40b krb5kdc[20987](info): commencing operation
krb5kdc: starting...
W20260927 16:42:52.055315 392 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:42:52 dist-test-slave-z40b krb5kdc[20987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:42:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:42:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:42:52.165930 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39717
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:42:52.265410 21000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:52.265655 21000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:52.265719 21000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:52.269588 21000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:42:52.269663 21000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:52.269697 21000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:52.269749 21000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:42:52.269785 21000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:42:52.274318 21000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:52.275152 21000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:52.276041 21000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:52.283941 21006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:52.283943 21008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:52.283955 21005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:52.284132 21000 server_base.cc:1034] running on GCE node
I20260927 16:42:52.284744 21000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:52.285760 21000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:52.286945 21000 hybrid_clock.cc:648] HybridClock initialized: now 1790527372286919 us; error 39 us; skew 500 ppm
Sep 27 16:42:52 dist-test-slave-z40b krb5kdc[20987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:52.295910 21000 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:42:52.296756 21000 webserver.cc:460] Webserver started at http://127.0.0.1:33209/ using document root <none> and password file <none>
I20260927 16:42:52.297300 21000 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:52.297389 21000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:52.297595 21000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:52.299602 21000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "fa3a3a81226f4807878433e536949106"
format_stamp: "Formatted at 2026-09-27 16:42:52 on dist-test-slave-z40b"
server_key: "03748cc3793566228e41dd11305bb709"
server_key_iv: "b4c218fc910e2329d67d40c7a4e0f73d"
server_key_version: "encryptionkey@0"
I20260927 16:42:52.300127 21000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "fa3a3a81226f4807878433e536949106"
format_stamp: "Formatted at 2026-09-27 16:42:52 on dist-test-slave-z40b"
server_key: "03748cc3793566228e41dd11305bb709"
server_key_iv: "b4c218fc910e2329d67d40c7a4e0f73d"
server_key_version: "encryptionkey@0"
I20260927 16:42:52.303586 21000 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 16:42:52.305606 21015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:52.306703 21000 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:42:52.306864 21000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "fa3a3a81226f4807878433e536949106"
format_stamp: "Formatted at 2026-09-27 16:42:52 on dist-test-slave-z40b"
server_key: "03748cc3793566228e41dd11305bb709"
server_key_iv: "b4c218fc910e2329d67d40c7a4e0f73d"
server_key_version: "encryptionkey@0"
I20260927 16:42:52.306978 21000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:52.325829 21000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:52.328434 21000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:52.328616 21000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:52.335044 21000 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39717
I20260927 16:42:52.335098 21067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39717 every 8 connection(s)
I20260927 16:42:52.335987 21000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:42:52.338485 21068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:52.341602 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 21000
I20260927 16:42:52.341771 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:42:52.342057 392 external_mini_cluster.cc:1428] Setting key 295ea6e9531f4c08a46bf73b1a719d23
I20260927 16:42:52.343979 21068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa3a3a81226f4807878433e536949106: Bootstrap starting.
I20260927 16:42:52.346215 21068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa3a3a81226f4807878433e536949106: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:52.346952 21068 log.cc:824] T 00000000000000000000000000000000 P fa3a3a81226f4807878433e536949106: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:52.348812 21068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa3a3a81226f4807878433e536949106: No bootstrap required, opened a new log
Sep 27 16:42:52 dist-test-slave-z40b krb5kdc[20987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 16:42:52.352520 21068 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa3a3a81226f4807878433e536949106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa3a3a81226f4807878433e536949106" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39717 } }
I20260927 16:42:52.352761 21068 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa3a3a81226f4807878433e536949106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:52.352847 21068 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa3a3a81226f4807878433e536949106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa3a3a81226f4807878433e536949106, State: Initialized, Role: FOLLOWER
I20260927 16:42:52.353298 21068 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa3a3a81226f4807878433e536949106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa3a3a81226f4807878433e536949106" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39717 } }
I20260927 16:42:52.353440 21068 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa3a3a81226f4807878433e536949106 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:42:52.353524 21068 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa3a3a81226f4807878433e536949106 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:42:52.353618 21068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa3a3a81226f4807878433e536949106 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:52.354400 21068 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa3a3a81226f4807878433e536949106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa3a3a81226f4807878433e536949106" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39717 } }
I20260927 16:42:52.354736 21068 leader_election.cc:302] T 00000000000000000000000000000000 P fa3a3a81226f4807878433e536949106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa3a3a81226f4807878433e536949106; no voters:
I20260927 16:42:52.355070 21068 leader_election.cc:288] T 00000000000000000000000000000000 P fa3a3a81226f4807878433e536949106 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:42:52.355134 21073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa3a3a81226f4807878433e536949106 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:42:52.355369 21073 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa3a3a81226f4807878433e536949106 [term 1 LEADER]: Becoming Leader. State: Replica: fa3a3a81226f4807878433e536949106, State: Running, Role: LEADER
I20260927 16:42:52.355640 21073 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa3a3a81226f4807878433e536949106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa3a3a81226f4807878433e536949106" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39717 } }
I20260927 16:42:52.355830 21071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:52.343327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48755 (local address 127.0.0.1:39717)
0927 16:42:52.343696 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:52.343711 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:52.343743 (+ 32us) server_negotiation.cc:407] Connection header received
0927 16:42:52.344408 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:52.344424 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:52.344730 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:52.344987 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:52.345804 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:52.346792 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:52.347865 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:52.348311 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:52.348424 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:52.351080 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:52.351085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:52.351096 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:52.351120 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:52.352959 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:52.353384 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:52.353389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:52.353395 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:52.353532 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:52.353805 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:52.353810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:52.353812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:52.354159 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:52.354354 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:52.354626 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 16:42:52.354926 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":119,"threads_started":1}
I20260927 16:42:52.356261 21068 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa3a3a81226f4807878433e536949106 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:42:52.357144 21074 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa3a3a81226f4807878433e536949106 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa3a3a81226f4807878433e536949106" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa3a3a81226f4807878433e536949106" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39717 } } }
I20260927 16:42:52.357260 21074 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa3a3a81226f4807878433e536949106 [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:52.357532 21075 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa3a3a81226f4807878433e536949106 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa3a3a81226f4807878433e536949106. Latest consensus state: current_term: 1 leader_uuid: "fa3a3a81226f4807878433e536949106" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa3a3a81226f4807878433e536949106" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39717 } } }
I20260927 16:42:52.357626 21082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:42:52.357612 21075 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa3a3a81226f4807878433e536949106 [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:52.358757 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:42:52.360708 21082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:42:52.365753 21082 catalog_manager.cc:1347] Generated new cluster ID: 77abab9fc4a04b0a866ba2eb1d92fb23
I20260927 16:42:52.365825 21082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:42:52.371068 21071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:52.361004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39605 (local address 127.0.0.1:39717)
0927 16:42:52.361094 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:52.361098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:52.361279 (+ 181us) server_negotiation.cc:407] Connection header received
0927 16:42:52.361596 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:52.361600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:52.361640 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:52.361720 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:52.362436 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:52.363133 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:52.364645 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:52.365013 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:52.365115 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:52.368100 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:52.368103 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:52.368106 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:52.368128 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:52.369506 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:52.369946 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:52.369951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:52.369953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:52.370026 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:52.370284 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:52.370288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:52.370290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:52.370449 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:52.370572 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:52.370795 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 16:42:52.370918 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 16:42:52.382397 21082 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:42:52.383282 21082 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:42:52.390818 21082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa3a3a81226f4807878433e536949106: Generated new TSK 0
I20260927 16:42:52.391553 21082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:42:52.403168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 21000
2026-09-27T16:42:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:42:52 dist-test-slave-z40b krb5kdc[21099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:42:52 dist-test-slave-z40b krb5kdc[21099](info): set up 2 sockets
Sep 27 16:42:52 dist-test-slave-z40b krb5kdc[21099](info): commencing operation
krb5kdc: starting...
W20260927 16:42:53.407430 392 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:42:53 dist-test-slave-z40b krb5kdc[21099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:42:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:42:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:42:53.517984 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44633
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:42:53.620337 21112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:53.620723 21112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:53.620792 21112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:53.625099 21112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:42:53.625152 21112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:53.625171 21112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:53.625186 21112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:42:53.625200 21112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:42:53.629987 21112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:53.630808 21112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:53.631845 21112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:53.639719 21120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:53.639729 21118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:53.639751 21117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:53.639905 21112 server_base.cc:1034] running on GCE node
I20260927 16:42:53.640565 21112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:53.641506 21112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:53.642656 21112 hybrid_clock.cc:648] HybridClock initialized: now 1790527373642640 us; error 27 us; skew 500 ppm
Sep 27 16:42:53 dist-test-slave-z40b krb5kdc[21099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:53.645212 21112 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:42:53.645965 21112 webserver.cc:460] Webserver started at http://127.0.0.1:41919/ using document root <none> and password file <none>
I20260927 16:42:53.646457 21112 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:53.646512 21112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:53.646723 21112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:53.648798 21112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "086e07596d764a3daeff0a6f3825a321"
format_stamp: "Formatted at 2026-09-27 16:42:53 on dist-test-slave-z40b"
server_key: "7fa8c7e04a956d06a40fae79a837e206"
server_key_iv: "87fe94320b1ad255620ec8087d9c92cc"
server_key_version: "encryptionkey@0"
I20260927 16:42:53.649336 21112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "086e07596d764a3daeff0a6f3825a321"
format_stamp: "Formatted at 2026-09-27 16:42:53 on dist-test-slave-z40b"
server_key: "7fa8c7e04a956d06a40fae79a837e206"
server_key_iv: "87fe94320b1ad255620ec8087d9c92cc"
server_key_version: "encryptionkey@0"
I20260927 16:42:53.652765 21112 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:42:53.654932 21127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:53.656095 21112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:42:53.656224 21112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "086e07596d764a3daeff0a6f3825a321"
format_stamp: "Formatted at 2026-09-27 16:42:53 on dist-test-slave-z40b"
server_key: "7fa8c7e04a956d06a40fae79a837e206"
server_key_iv: "87fe94320b1ad255620ec8087d9c92cc"
server_key_version: "encryptionkey@0"
I20260927 16:42:53.656327 21112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:53.668172 21112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:53.670600 21112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:53.670778 21112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:53.677464 21112 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44633
I20260927 16:42:53.677480 21179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44633 every 8 connection(s)
I20260927 16:42:53.678323 21112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:42:53.680922 21180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:53.682924 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 21112
I20260927 16:42:53.683101 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:42:53.683409 392 external_mini_cluster.cc:1428] Setting key 5582edca60bf472c8e258453821dc82c
I20260927 16:42:53.686483 21180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 086e07596d764a3daeff0a6f3825a321: Bootstrap starting.
I20260927 16:42:53.688635 21180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 086e07596d764a3daeff0a6f3825a321: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:53.689320 21180 log.cc:824] T 00000000000000000000000000000000 P 086e07596d764a3daeff0a6f3825a321: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:53.691483 21180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 086e07596d764a3daeff0a6f3825a321: No bootstrap required, opened a new log
I20260927 16:42:53.692085 21183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:53.684697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44595 (local address 127.0.0.1:44633)
0927 16:42:53.685097 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:53.685110 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:53.685141 (+ 31us) server_negotiation.cc:407] Connection header received
0927 16:42:53.685721 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:53.685735 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:53.686050 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:53.686363 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:53.686743 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:53.687784 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:53.688668 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:53.689154 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:53.689317 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:53.690174 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:53.690182 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:53.690200 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:42:53.690678 (+ 478us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:53.690741 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:42:53.690775 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:53.690877 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:53.690980 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 16:42:53.691163 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":126,"threads_started":1}
I20260927 16:42:53.695034 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:42:53.694927 21180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 086e07596d764a3daeff0a6f3825a321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086e07596d764a3daeff0a6f3825a321" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44633 } }
I20260927 16:42:53.695181 21180 raft_consensus.cc:374] T 00000000000000000000000000000000 P 086e07596d764a3daeff0a6f3825a321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:53.695250 21180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 086e07596d764a3daeff0a6f3825a321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 086e07596d764a3daeff0a6f3825a321, State: Initialized, Role: FOLLOWER
I20260927 16:42:53.695685 21180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 086e07596d764a3daeff0a6f3825a321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086e07596d764a3daeff0a6f3825a321" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44633 } }
I20260927 16:42:53.695828 21180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 086e07596d764a3daeff0a6f3825a321 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:42:53.695935 21180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 086e07596d764a3daeff0a6f3825a321 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:42:53.696029 21180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 086e07596d764a3daeff0a6f3825a321 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:53.697125 21180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 086e07596d764a3daeff0a6f3825a321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086e07596d764a3daeff0a6f3825a321" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44633 } }
I20260927 16:42:53.697471 21180 leader_election.cc:302] T 00000000000000000000000000000000 P 086e07596d764a3daeff0a6f3825a321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 086e07596d764a3daeff0a6f3825a321; no voters:
I20260927 16:42:53.697810 21180 leader_election.cc:288] T 00000000000000000000000000000000 P 086e07596d764a3daeff0a6f3825a321 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:42:53.697901 21190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 086e07596d764a3daeff0a6f3825a321 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:42:53.698086 21190 raft_consensus.cc:686] T 00000000000000000000000000000000 P 086e07596d764a3daeff0a6f3825a321 [term 1 LEADER]: Becoming Leader. State: Replica: 086e07596d764a3daeff0a6f3825a321, State: Running, Role: LEADER
I20260927 16:42:53.698378 21190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 086e07596d764a3daeff0a6f3825a321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086e07596d764a3daeff0a6f3825a321" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44633 } }
I20260927 16:42:53.699045 21180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 086e07596d764a3daeff0a6f3825a321 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:42:53.699528 21191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 086e07596d764a3daeff0a6f3825a321 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 086e07596d764a3daeff0a6f3825a321. Latest consensus state: current_term: 1 leader_uuid: "086e07596d764a3daeff0a6f3825a321" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086e07596d764a3daeff0a6f3825a321" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44633 } } }
I20260927 16:42:53.699730 21191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 086e07596d764a3daeff0a6f3825a321 [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:53.699556 21192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 086e07596d764a3daeff0a6f3825a321 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "086e07596d764a3daeff0a6f3825a321" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086e07596d764a3daeff0a6f3825a321" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44633 } } }
I20260927 16:42:53.699869 21192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 086e07596d764a3daeff0a6f3825a321 [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:53.700299 21198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:42:53.701646 21183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:53.697246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38477 (local address 127.0.0.1:44633)
0927 16:42:53.697348 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:53.697353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:53.697619 (+ 266us) server_negotiation.cc:407] Connection header received
0927 16:42:53.697707 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:53.697709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:53.697744 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:53.697866 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:53.698080 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:53.698680 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:53.699716 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:53.700086 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:53.700204 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:53.701102 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:53.701106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:53.701108 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:42:53.701119 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:53.701143 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:42:53.701154 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:53.701210 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:53.701488 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 16:42:53.701533 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 16:42:53.703342 21198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:42:53.708670 21198 catalog_manager.cc:1347] Generated new cluster ID: 817cb17e64004fcf8974ef6bb6fd53cc
I20260927 16:42:53.708761 21198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:42:53.726325 21198 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:42:53.727195 21198 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:42:53.734668 21198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 086e07596d764a3daeff0a6f3825a321: Generated new TSK 0
I20260927 16:42:53.735376 21198 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:42:53.744849 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 21112
2026-09-27T16:42:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:42:53 dist-test-slave-z40b krb5kdc[21211](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:42:53 dist-test-slave-z40b krb5kdc[21211](info): set up 2 sockets
Sep 27 16:42:53 dist-test-slave-z40b krb5kdc[21211](info): commencing operation
krb5kdc: starting...
W20260927 16:42:54.747251 392 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:42:54 dist-test-slave-z40b krb5kdc[21211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:42:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:42:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:42:54.862985 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46619
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:42:54.964849 21224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:54.965057 21224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:54.965092 21224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:54.969002 21224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:42:54.969048 21224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:54.969066 21224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:54.969082 21224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:42:54.969131 21224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:42:54.973899 21224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:54.974762 21224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:54.975762 21224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:54.983632 21230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:54.983639 21232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:54.983692 21224 server_base.cc:1034] running on GCE node
W20260927 16:42:54.983639 21229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:54.984403 21224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:54.985389 21224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:54.986570 21224 hybrid_clock.cc:648] HybridClock initialized: now 1790527374986553 us; error 28 us; skew 500 ppm
Sep 27 16:42:54 dist-test-slave-z40b krb5kdc[21211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:54.989283 21224 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:42:54.990036 21224 webserver.cc:460] Webserver started at http://127.0.0.1:35821/ using document root <none> and password file <none>
I20260927 16:42:54.990516 21224 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:54.990568 21224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:54.990713 21224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:54.992540 21224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "fec6db567f5d427c90e5bfe85ba983a5"
format_stamp: "Formatted at 2026-09-27 16:42:54 on dist-test-slave-z40b"
server_key: "936669df67b0113589355a3f91781106"
server_key_iv: "77d8e8f6dadc30e8ee3408a5cdff7c51"
server_key_version: "encryptionkey@0"
I20260927 16:42:54.993007 21224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "fec6db567f5d427c90e5bfe85ba983a5"
format_stamp: "Formatted at 2026-09-27 16:42:54 on dist-test-slave-z40b"
server_key: "936669df67b0113589355a3f91781106"
server_key_iv: "77d8e8f6dadc30e8ee3408a5cdff7c51"
server_key_version: "encryptionkey@0"
I20260927 16:42:54.996397 21224 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:42:54.998605 21239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:54.999782 21224 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:42:54.999914 21224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "fec6db567f5d427c90e5bfe85ba983a5"
format_stamp: "Formatted at 2026-09-27 16:42:54 on dist-test-slave-z40b"
server_key: "936669df67b0113589355a3f91781106"
server_key_iv: "77d8e8f6dadc30e8ee3408a5cdff7c51"
server_key_version: "encryptionkey@0"
I20260927 16:42:55.000015 21224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:55.004348 21224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:55.007098 21224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:55.007341 21224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:55.013856 21224 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46619
I20260927 16:42:55.013911 21291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46619 every 8 connection(s)
I20260927 16:42:55.014696 21224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:42:55.017259 21292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:55.018208 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 21224
I20260927 16:42:55.018381 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:42:55.018646 392 external_mini_cluster.cc:1428] Setting key b94c43f54d9a3b1fa31f7015bb523b2c
I20260927 16:42:55.023142 21292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fec6db567f5d427c90e5bfe85ba983a5: Bootstrap starting.
I20260927 16:42:55.023440 21295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:55.019874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42101 (local address 127.0.0.1:46619)
0927 16:42:55.020230 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:55.020243 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:55.020275 (+ 32us) server_negotiation.cc:407] Connection header received
0927 16:42:55.020891 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:55.020907 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:55.021175 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:55.021443 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:55.022265 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:55.022269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:55.022279 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:42:55.022288 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:55.022335 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:42:55.022359 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:55.022461 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:55.022579 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 16:42:55.022729 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":116,"threads_started":1}
I20260927 16:42:55.025431 21292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fec6db567f5d427c90e5bfe85ba983a5: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:55.026048 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:42:55.026291 21292 log.cc:824] T 00000000000000000000000000000000 P fec6db567f5d427c90e5bfe85ba983a5: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:55.028429 21292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fec6db567f5d427c90e5bfe85ba983a5: No bootstrap required, opened a new log
I20260927 16:42:55.029717 21295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:55.028237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44873 (local address 127.0.0.1:46619)
0927 16:42:55.028342 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:55.028345 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:55.028509 (+ 164us) server_negotiation.cc:407] Connection header received
0927 16:42:55.028618 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:55.028622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:55.028653 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:55.028722 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:55.029333 (+ 611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:55.029336 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:55.029337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:42:55.029355 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:55.029375 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:42:55.029384 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:55.029448 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:55.029555 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 16:42:55.029582 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 16:42:55.032261 21292 raft_consensus.cc:348] T 00000000000000000000000000000000 P fec6db567f5d427c90e5bfe85ba983a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec6db567f5d427c90e5bfe85ba983a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46619 } }
I20260927 16:42:55.032450 21292 raft_consensus.cc:374] T 00000000000000000000000000000000 P fec6db567f5d427c90e5bfe85ba983a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:55.032536 21292 raft_consensus.cc:729] T 00000000000000000000000000000000 P fec6db567f5d427c90e5bfe85ba983a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec6db567f5d427c90e5bfe85ba983a5, State: Initialized, Role: FOLLOWER
I20260927 16:42:55.032927 21292 consensus_queue.cc:260] T 00000000000000000000000000000000 P fec6db567f5d427c90e5bfe85ba983a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec6db567f5d427c90e5bfe85ba983a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46619 } }
I20260927 16:42:55.033085 21292 raft_consensus.cc:388] T 00000000000000000000000000000000 P fec6db567f5d427c90e5bfe85ba983a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:42:55.033156 21292 raft_consensus.cc:482] T 00000000000000000000000000000000 P fec6db567f5d427c90e5bfe85ba983a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:42:55.033249 21292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fec6db567f5d427c90e5bfe85ba983a5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:55.034056 21292 raft_consensus.cc:504] T 00000000000000000000000000000000 P fec6db567f5d427c90e5bfe85ba983a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec6db567f5d427c90e5bfe85ba983a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46619 } }
I20260927 16:42:55.034358 21292 leader_election.cc:302] T 00000000000000000000000000000000 P fec6db567f5d427c90e5bfe85ba983a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fec6db567f5d427c90e5bfe85ba983a5; no voters:
I20260927 16:42:55.034621 21292 leader_election.cc:288] T 00000000000000000000000000000000 P fec6db567f5d427c90e5bfe85ba983a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:42:55.034723 21302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fec6db567f5d427c90e5bfe85ba983a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:42:55.034968 21302 raft_consensus.cc:686] T 00000000000000000000000000000000 P fec6db567f5d427c90e5bfe85ba983a5 [term 1 LEADER]: Becoming Leader. State: Replica: fec6db567f5d427c90e5bfe85ba983a5, State: Running, Role: LEADER
I20260927 16:42:55.035346 21302 consensus_queue.cc:237] T 00000000000000000000000000000000 P fec6db567f5d427c90e5bfe85ba983a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec6db567f5d427c90e5bfe85ba983a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46619 } }
I20260927 16:42:55.035756 21292 sys_catalog.cc:564] T 00000000000000000000000000000000 P fec6db567f5d427c90e5bfe85ba983a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:42:55.036834 21304 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec6db567f5d427c90e5bfe85ba983a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fec6db567f5d427c90e5bfe85ba983a5. Latest consensus state: current_term: 1 leader_uuid: "fec6db567f5d427c90e5bfe85ba983a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec6db567f5d427c90e5bfe85ba983a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46619 } } }
I20260927 16:42:55.036952 21304 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec6db567f5d427c90e5bfe85ba983a5 [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:55.037276 21303 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec6db567f5d427c90e5bfe85ba983a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fec6db567f5d427c90e5bfe85ba983a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec6db567f5d427c90e5bfe85ba983a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46619 } } }
I20260927 16:42:55.037330 21311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:42:55.037873 21303 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec6db567f5d427c90e5bfe85ba983a5 [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:55.040535 21311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:42:55.045032 21311 catalog_manager.cc:1347] Generated new cluster ID: 795a0cf6cc2947278f3e3231491abe6a
I20260927 16:42:55.045104 21311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:42:55.063968 21311 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:42:55.064239 21311 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:42:55.072817 21311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fec6db567f5d427c90e5bfe85ba983a5: Generated new TSK 0
I20260927 16:42:55.073483 21311 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:42:55.077627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 21224
2026-09-27T16:42:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:42:55 dist-test-slave-z40b krb5kdc[21323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:42:55 dist-test-slave-z40b krb5kdc[21323](info): set up 2 sockets
Sep 27 16:42:55 dist-test-slave-z40b krb5kdc[21323](info): commencing operation
krb5kdc: starting...
W20260927 16:42:56.079531 392 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:42:56 dist-test-slave-z40b krb5kdc[21323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:42:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:42:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:42:56.190107 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:42:56.292335 21336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:56.292562 21336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:56.292600 21336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:56.296556 21336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:42:56.296633 21336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:56.296669 21336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:56.296725 21336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:42:56.296763 21336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:42:56.301657 21336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:56.302582 21336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:56.303519 21336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:56.311517 21344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:56.311506 21341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:56.311537 21342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:56.311713 21336 server_base.cc:1034] running on GCE node
I20260927 16:42:56.312523 21336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:56.313558 21336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:56.314741 21336 hybrid_clock.cc:648] HybridClock initialized: now 1790527376314734 us; error 39 us; skew 500 ppm
Sep 27 16:42:56 dist-test-slave-z40b krb5kdc[21323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:56.317327 21336 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:42:56.318117 21336 webserver.cc:460] Webserver started at http://127.0.0.1:40615/ using document root <none> and password file <none>
I20260927 16:42:56.318652 21336 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:56.318758 21336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:56.318975 21336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:56.320915 21336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "616ec68956aa43a0b2f6b1431c626e61"
format_stamp: "Formatted at 2026-09-27 16:42:56 on dist-test-slave-z40b"
server_key: "bc8f35f4542336b025cdb1b298a24aa8"
server_key_iv: "ec0a69b98c8fe340a981b0e41c51a03b"
server_key_version: "encryptionkey@0"
I20260927 16:42:56.321465 21336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "616ec68956aa43a0b2f6b1431c626e61"
format_stamp: "Formatted at 2026-09-27 16:42:56 on dist-test-slave-z40b"
server_key: "bc8f35f4542336b025cdb1b298a24aa8"
server_key_iv: "ec0a69b98c8fe340a981b0e41c51a03b"
server_key_version: "encryptionkey@0"
I20260927 16:42:56.324816 21336 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260927 16:42:56.326954 21351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:56.328088 21336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:42:56.328223 21336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "616ec68956aa43a0b2f6b1431c626e61"
format_stamp: "Formatted at 2026-09-27 16:42:56 on dist-test-slave-z40b"
server_key: "bc8f35f4542336b025cdb1b298a24aa8"
server_key_iv: "ec0a69b98c8fe340a981b0e41c51a03b"
server_key_version: "encryptionkey@0"
I20260927 16:42:56.328330 21336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:56.346017 21336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:56.348629 21336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:56.348814 21336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:56.355075 21336 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44315
I20260927 16:42:56.355098 21403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44315 every 8 connection(s)
I20260927 16:42:56.355927 21336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:42:56.358417 21404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:56.363466 21404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 616ec68956aa43a0b2f6b1431c626e61: Bootstrap starting.
I20260927 16:42:56.365114 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 21336
I20260927 16:42:56.365254 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:42:56.365558 392 external_mini_cluster.cc:1428] Setting key 96a51fde7e091c9a0fe79b98b2886082
I20260927 16:42:56.365788 21404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 616ec68956aa43a0b2f6b1431c626e61: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:56.366559 21404 log.cc:824] T 00000000000000000000000000000000 P 616ec68956aa43a0b2f6b1431c626e61: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:56.368357 21404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 616ec68956aa43a0b2f6b1431c626e61: No bootstrap required, opened a new log
I20260927 16:42:56.371546 21404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 616ec68956aa43a0b2f6b1431c626e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616ec68956aa43a0b2f6b1431c626e61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44315 } }
I20260927 16:42:56.371762 21404 raft_consensus.cc:374] T 00000000000000000000000000000000 P 616ec68956aa43a0b2f6b1431c626e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:56.371819 21404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 616ec68956aa43a0b2f6b1431c626e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616ec68956aa43a0b2f6b1431c626e61, State: Initialized, Role: FOLLOWER
I20260927 16:42:56.372267 21404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 616ec68956aa43a0b2f6b1431c626e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616ec68956aa43a0b2f6b1431c626e61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44315 } }
I20260927 16:42:56.372418 21404 raft_consensus.cc:388] T 00000000000000000000000000000000 P 616ec68956aa43a0b2f6b1431c626e61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:42:56.372502 21404 raft_consensus.cc:482] T 00000000000000000000000000000000 P 616ec68956aa43a0b2f6b1431c626e61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:42:56.372603 21404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 616ec68956aa43a0b2f6b1431c626e61 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:56.373561 21404 raft_consensus.cc:504] T 00000000000000000000000000000000 P 616ec68956aa43a0b2f6b1431c626e61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616ec68956aa43a0b2f6b1431c626e61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44315 } }
Sep 27 16:42:56 dist-test-slave-z40b krb5kdc[21323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 16:42:56.373903 21404 leader_election.cc:302] T 00000000000000000000000000000000 P 616ec68956aa43a0b2f6b1431c626e61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 616ec68956aa43a0b2f6b1431c626e61; no voters:
I20260927 16:42:56.374221 21404 leader_election.cc:288] T 00000000000000000000000000000000 P 616ec68956aa43a0b2f6b1431c626e61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:42:56.374325 21409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 616ec68956aa43a0b2f6b1431c626e61 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:42:56.374569 21409 raft_consensus.cc:686] T 00000000000000000000000000000000 P 616ec68956aa43a0b2f6b1431c626e61 [term 1 LEADER]: Becoming Leader. State: Replica: 616ec68956aa43a0b2f6b1431c626e61, State: Running, Role: LEADER
I20260927 16:42:56.375259 21409 consensus_queue.cc:237] T 00000000000000000000000000000000 P 616ec68956aa43a0b2f6b1431c626e61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616ec68956aa43a0b2f6b1431c626e61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44315 } }
I20260927 16:42:56.375483 21404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 616ec68956aa43a0b2f6b1431c626e61 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:42:56.376372 21409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 616ec68956aa43a0b2f6b1431c626e61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 616ec68956aa43a0b2f6b1431c626e61. Latest consensus state: current_term: 1 leader_uuid: "616ec68956aa43a0b2f6b1431c626e61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616ec68956aa43a0b2f6b1431c626e61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44315 } } }
I20260927 16:42:56.376579 21409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 616ec68956aa43a0b2f6b1431c626e61 [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:56.376801 21410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 616ec68956aa43a0b2f6b1431c626e61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "616ec68956aa43a0b2f6b1431c626e61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616ec68956aa43a0b2f6b1431c626e61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44315 } } }
I20260927 16:42:56.376889 21410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 616ec68956aa43a0b2f6b1431c626e61 [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:56.377028 21418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:42:56.380584 21407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:56.366585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:52127 (local address 127.0.0.1:44315)
0927 16:42:56.367041 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:56.367062 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:56.367103 (+ 41us) server_negotiation.cc:407] Connection header received
0927 16:42:56.367780 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:56.367796 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:56.368110 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:56.368443 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:56.369193 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:56.370090 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:56.370915 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:56.371431 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:56.371592 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:56.374487 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:56.374493 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:56.374510 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:56.374546 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:56.376711 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:56.377360 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:56.377366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:56.377371 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:56.377498 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:56.377864 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:56.377867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:56.377869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:56.378337 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:56.379275 (+ 938us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:56.379456 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 16:42:56.379689 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":140,"threads_started":1}
I20260927 16:42:56.380944 21418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:42:56.383440 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:42:56.386435 21418 catalog_manager.cc:1347] Generated new cluster ID: 7a7f6b575daf4b0daabf84a506a4f466
I20260927 16:42:56.386502 21418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:42:56.394980 21407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:56.385310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:56495 (local address 127.0.0.1:44315)
0927 16:42:56.385415 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:56.385419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:56.386852 (+ 1433us) server_negotiation.cc:407] Connection header received
0927 16:42:56.386956 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:56.386959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:56.387002 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:56.387084 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:56.387892 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:56.388640 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:56.389845 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:56.390180 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:56.390265 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:56.391862 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:56.391866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:56.391868 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:42:56.391889 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:56.393404 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:56.393895 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:56.393898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:56.393900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:56.393965 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:42:56.394261 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:42:56.394263 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:42:56.394264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:42:56.394420 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:56.394539 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:56.394777 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 16:42:56.394857 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:42:56.401121 21418 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:42:56.401836 21418 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:42:56.407262 21418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 616ec68956aa43a0b2f6b1431c626e61: Generated new TSK 0
I20260927 16:42:56.407857 21418 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:42:56.414180 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 21336
W20260927 16:42:56.418780 21321 connection.cc:535] client connection to 127.0.0.1:44315 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44315): Connection reset by peer (error 104)
2026-09-27T16:42:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:42:56 dist-test-slave-z40b krb5kdc[21435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:42:56 dist-test-slave-z40b krb5kdc[21435](info): set up 2 sockets
Sep 27 16:42:56 dist-test-slave-z40b krb5kdc[21435](info): commencing operation
krb5kdc: starting...
W20260927 16:42:57.415269 392 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:42:57 dist-test-slave-z40b krb5kdc[21435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:42:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:42:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:42:57.526126 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37531
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:42:57.626060 21448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:57.626314 21448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:57.626385 21448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:57.630183 21448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:42:57.630260 21448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:57.630295 21448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:57.630328 21448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:42:57.630360 21448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:42:57.636188 21448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:57.637079 21448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:57.638006 21448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:57.645522 21456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:57.645546 21454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:57.645572 21448 server_base.cc:1034] running on GCE node
W20260927 16:42:57.645803 21453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:57.646396 21448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:57.647472 21448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:57.648624 21448 hybrid_clock.cc:648] HybridClock initialized: now 1790527377648599 us; error 37 us; skew 500 ppm
Sep 27 16:42:57 dist-test-slave-z40b krb5kdc[21435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:57.656787 21448 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:42:57.657696 21448 webserver.cc:460] Webserver started at http://127.0.0.1:43581/ using document root <none> and password file <none>
I20260927 16:42:57.658222 21448 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:57.658311 21448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:57.658550 21448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:57.660487 21448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "fef0ba04894546a192e10d150702fc79"
format_stamp: "Formatted at 2026-09-27 16:42:57 on dist-test-slave-z40b"
server_key: "5dda2c00b15c57d98738682e4a609076"
server_key_iv: "cc1700eb9fa7364ab1f81c5c6a4f6bf5"
server_key_version: "encryptionkey@0"
I20260927 16:42:57.661001 21448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "fef0ba04894546a192e10d150702fc79"
format_stamp: "Formatted at 2026-09-27 16:42:57 on dist-test-slave-z40b"
server_key: "5dda2c00b15c57d98738682e4a609076"
server_key_iv: "cc1700eb9fa7364ab1f81c5c6a4f6bf5"
server_key_version: "encryptionkey@0"
I20260927 16:42:57.664274 21448 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:42:57.666498 21463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:57.667680 21448 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:42:57.667820 21448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "fef0ba04894546a192e10d150702fc79"
format_stamp: "Formatted at 2026-09-27 16:42:57 on dist-test-slave-z40b"
server_key: "5dda2c00b15c57d98738682e4a609076"
server_key_iv: "cc1700eb9fa7364ab1f81c5c6a4f6bf5"
server_key_version: "encryptionkey@0"
I20260927 16:42:57.667927 21448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:57.685087 21448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:57.687712 21448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:57.687906 21448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:57.694356 21448 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37531
I20260927 16:42:57.694360 21515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37531 every 8 connection(s)
I20260927 16:42:57.695190 21448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:42:57.697687 21516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:57.701515 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 21448
I20260927 16:42:57.701644 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:42:57.701908 392 external_mini_cluster.cc:1428] Setting key 77f0062a9b767df3ad124204604aba5c
I20260927 16:42:57.702868 21516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fef0ba04894546a192e10d150702fc79: Bootstrap starting.
I20260927 16:42:57.704982 21516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fef0ba04894546a192e10d150702fc79: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:57.705703 21516 log.cc:824] T 00000000000000000000000000000000 P fef0ba04894546a192e10d150702fc79: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:57.707818 21516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fef0ba04894546a192e10d150702fc79: No bootstrap required, opened a new log
I20260927 16:42:57.710598 21519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:57.703141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:39151 (local address 127.0.0.1:37531)
0927 16:42:57.703548 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:57.703563 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:57.703594 (+ 31us) server_negotiation.cc:407] Connection header received
0927 16:42:57.704144 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:57.704158 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:57.704477 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:57.704726 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:57.705142 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:57.706269 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:57.707588 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:57.708081 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:57.708193 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:57.709029 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:57.709034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:57.709045 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:42:57.709418 (+ 373us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:57.709462 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:42:57.709481 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:57.709576 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:57.709804 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 16:42:57.709944 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":121,"threads_started":1}
I20260927 16:42:57.711743 21516 raft_consensus.cc:348] T 00000000000000000000000000000000 P fef0ba04894546a192e10d150702fc79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef0ba04894546a192e10d150702fc79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37531 } }
I20260927 16:42:57.711995 21516 raft_consensus.cc:374] T 00000000000000000000000000000000 P fef0ba04894546a192e10d150702fc79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:57.712069 21516 raft_consensus.cc:729] T 00000000000000000000000000000000 P fef0ba04894546a192e10d150702fc79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef0ba04894546a192e10d150702fc79, State: Initialized, Role: FOLLOWER
I20260927 16:42:57.712474 21516 consensus_queue.cc:260] T 00000000000000000000000000000000 P fef0ba04894546a192e10d150702fc79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef0ba04894546a192e10d150702fc79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37531 } }
I20260927 16:42:57.712618 21516 raft_consensus.cc:388] T 00000000000000000000000000000000 P fef0ba04894546a192e10d150702fc79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:42:57.712682 21516 raft_consensus.cc:482] T 00000000000000000000000000000000 P fef0ba04894546a192e10d150702fc79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:42:57.712801 21516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fef0ba04894546a192e10d150702fc79 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:57.713328 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:42:57.713672 21516 raft_consensus.cc:504] T 00000000000000000000000000000000 P fef0ba04894546a192e10d150702fc79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef0ba04894546a192e10d150702fc79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37531 } }
I20260927 16:42:57.713987 21516 leader_election.cc:302] T 00000000000000000000000000000000 P fef0ba04894546a192e10d150702fc79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fef0ba04894546a192e10d150702fc79; no voters:
I20260927 16:42:57.714277 21516 leader_election.cc:288] T 00000000000000000000000000000000 P fef0ba04894546a192e10d150702fc79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:42:57.714371 21521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fef0ba04894546a192e10d150702fc79 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:42:57.714562 21521 raft_consensus.cc:686] T 00000000000000000000000000000000 P fef0ba04894546a192e10d150702fc79 [term 1 LEADER]: Becoming Leader. State: Replica: fef0ba04894546a192e10d150702fc79, State: Running, Role: LEADER
I20260927 16:42:57.714943 21521 consensus_queue.cc:237] T 00000000000000000000000000000000 P fef0ba04894546a192e10d150702fc79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef0ba04894546a192e10d150702fc79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37531 } }
I20260927 16:42:57.715612 21516 sys_catalog.cc:564] T 00000000000000000000000000000000 P fef0ba04894546a192e10d150702fc79 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:42:57.716164 21525 sys_catalog.cc:455] T 00000000000000000000000000000000 P fef0ba04894546a192e10d150702fc79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fef0ba04894546a192e10d150702fc79. Latest consensus state: current_term: 1 leader_uuid: "fef0ba04894546a192e10d150702fc79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef0ba04894546a192e10d150702fc79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37531 } } }
I20260927 16:42:57.716296 21525 sys_catalog.cc:458] T 00000000000000000000000000000000 P fef0ba04894546a192e10d150702fc79 [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:57.716734 21533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:42:57.717936 21523 sys_catalog.cc:455] T 00000000000000000000000000000000 P fef0ba04894546a192e10d150702fc79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fef0ba04894546a192e10d150702fc79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef0ba04894546a192e10d150702fc79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37531 } } }
I20260927 16:42:57.718087 21523 sys_catalog.cc:458] T 00000000000000000000000000000000 P fef0ba04894546a192e10d150702fc79 [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:57.719988 21533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:42:57.721060 21519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:57.715422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:47115 (local address 127.0.0.1:37531)
0927 16:42:57.715520 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:57.715524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:57.715813 (+ 289us) server_negotiation.cc:407] Connection header received
0927 16:42:57.715919 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:57.715922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:57.715972 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:57.716056 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:57.716469 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:57.717215 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:57.718469 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:42:57.718761 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:42:57.718872 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:42:57.720591 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:57.720595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:57.720597 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:42:57.720611 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:57.720636 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:42:57.720646 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:57.720748 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:57.720906 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 16:42:57.720935 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 16:42:57.725628 21533 catalog_manager.cc:1347] Generated new cluster ID: f3a8b093c2234dad814f3c34e18e187a
I20260927 16:42:57.725698 21533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:42:57.734719 21533 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:42:57.735561 21533 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:42:57.741544 21533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fef0ba04894546a192e10d150702fc79: Generated new TSK 0
I20260927 16:42:57.742103 21533 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:42:57.747164 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 21448
W20260927 16:42:57.752599 21433 connection.cc:535] client connection to 127.0.0.1:37531 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37531): Connection reset by peer (error 104)
2026-09-27T16:42:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:42:57 dist-test-slave-z40b krb5kdc[21547](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:42:57 dist-test-slave-z40b krb5kdc[21547](info): set up 2 sockets
Sep 27 16:42:57 dist-test-slave-z40b krb5kdc[21547](info): commencing operation
krb5kdc: starting...
W20260927 16:42:58.747440 392 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:42:58 dist-test-slave-z40b krb5kdc[21547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:42:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:42:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:42:58.862120 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37437
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:42:58.967059 21560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:42:58.967308 21560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:42:58.967348 21560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:42:58.971194 21560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:42:58.971257 21560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:42:58.971275 21560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:42:58.971325 21560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:42:58.971355 21560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:42:58.976231 21560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:42:58.977137 21560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:42:58.978125 21560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:42:58.986267 21568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:58.986301 21565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:42:58.986327 21566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:42:58.986378 21560 server_base.cc:1034] running on GCE node
I20260927 16:42:58.987133 21560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:42:58.988198 21560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:42:58.989360 21560 hybrid_clock.cc:648] HybridClock initialized: now 1790527378989333 us; error 44 us; skew 500 ppm
Sep 27 16:42:58 dist-test-slave-z40b krb5kdc[21547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:42:58.992111 21560 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:42:58.992995 21560 webserver.cc:460] Webserver started at http://127.0.0.1:40243/ using document root <none> and password file <none>
I20260927 16:42:58.993521 21560 fs_manager.cc:359] Metadata directory not provided
I20260927 16:42:58.993582 21560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:42:58.993791 21560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:42:58.995769 21560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "29f1e7b3165d465c861a7d8d41e76842"
format_stamp: "Formatted at 2026-09-27 16:42:58 on dist-test-slave-z40b"
server_key: "312d9383760fc26f13cc2a1bc7ad0afb"
server_key_iv: "0e4661dbb6f2818b6492a0bbca4a33fb"
server_key_version: "encryptionkey@0"
I20260927 16:42:58.996294 21560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "29f1e7b3165d465c861a7d8d41e76842"
format_stamp: "Formatted at 2026-09-27 16:42:58 on dist-test-slave-z40b"
server_key: "312d9383760fc26f13cc2a1bc7ad0afb"
server_key_iv: "0e4661dbb6f2818b6492a0bbca4a33fb"
server_key_version: "encryptionkey@0"
I20260927 16:42:58.999862 21560 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 16:42:59.002159 21575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:42:59.003412 21560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:42:59.003540 21560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "29f1e7b3165d465c861a7d8d41e76842"
format_stamp: "Formatted at 2026-09-27 16:42:58 on dist-test-slave-z40b"
server_key: "312d9383760fc26f13cc2a1bc7ad0afb"
server_key_iv: "0e4661dbb6f2818b6492a0bbca4a33fb"
server_key_version: "encryptionkey@0"
I20260927 16:42:59.003650 21560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:42:59.007843 21560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:42:59.016616 21560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:42:59.016803 21560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:42:59.023487 21560 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37437
I20260927 16:42:59.023509 21627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37437 every 8 connection(s)
I20260927 16:42:59.024310 21560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:42:59.026813 21628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:42:59.027396 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 21560
I20260927 16:42:59.027535 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:42:59.027808 392 external_mini_cluster.cc:1428] Setting key 1b07b9a95c25e84539e60031ed8720d1
I20260927 16:42:59.033047 21631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:59.029121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:45949 (local address 127.0.0.1:37437)
0927 16:42:59.029509 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:59.029527 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:59.029568 (+ 41us) server_negotiation.cc:407] Connection header received
0927 16:42:59.030377 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:59.030399 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:59.030760 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:59.031116 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:59.031791 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:59.031796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:59.031810 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:42:59.031820 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:59.031884 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:42:59.031910 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:59.032037 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:59.032100 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 16:42:59.032245 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":97,"threads_started":1}
I20260927 16:42:59.033286 21628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f1e7b3165d465c861a7d8d41e76842: Bootstrap starting.
I20260927 16:42:59.035758 21628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29f1e7b3165d465c861a7d8d41e76842: Neither blocks nor log segments found. Creating new log.
I20260927 16:42:59.035769 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:42:59.036558 21628 log.cc:824] T 00000000000000000000000000000000 P 29f1e7b3165d465c861a7d8d41e76842: Log is configured to *not* fsync() on all Append() calls
I20260927 16:42:59.038653 21628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f1e7b3165d465c861a7d8d41e76842: No bootstrap required, opened a new log
I20260927 16:42:59.039377 21631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:42:59.037906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:45071 (local address 127.0.0.1:37437)
0927 16:42:59.038023 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 16:42:59.038028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:42:59.038236 (+ 208us) server_negotiation.cc:407] Connection header received
0927 16:42:59.038298 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:42:59.038301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:42:59.038332 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:42:59.038415 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:42:59.039011 (+ 596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:42:59.039014 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:42:59.039016 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:42:59.039028 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:42:59.039052 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:42:59.039063 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:42:59.039147 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 16:42:59.039223 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 16:42:59.039268 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:42:59.042146 21628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29f1e7b3165d465c861a7d8d41e76842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f1e7b3165d465c861a7d8d41e76842" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37437 } }
I20260927 16:42:59.042353 21628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29f1e7b3165d465c861a7d8d41e76842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:42:59.042393 21628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29f1e7b3165d465c861a7d8d41e76842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f1e7b3165d465c861a7d8d41e76842, State: Initialized, Role: FOLLOWER
I20260927 16:42:59.042726 21628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29f1e7b3165d465c861a7d8d41e76842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f1e7b3165d465c861a7d8d41e76842" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37437 } }
I20260927 16:42:59.042826 21628 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29f1e7b3165d465c861a7d8d41e76842 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:42:59.042869 21628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29f1e7b3165d465c861a7d8d41e76842 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:42:59.042934 21628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29f1e7b3165d465c861a7d8d41e76842 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:42:59.043735 21628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29f1e7b3165d465c861a7d8d41e76842 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f1e7b3165d465c861a7d8d41e76842" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37437 } }
I20260927 16:42:59.044001 21628 leader_election.cc:302] T 00000000000000000000000000000000 P 29f1e7b3165d465c861a7d8d41e76842 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29f1e7b3165d465c861a7d8d41e76842; no voters:
I20260927 16:42:59.044215 21628 leader_election.cc:288] T 00000000000000000000000000000000 P 29f1e7b3165d465c861a7d8d41e76842 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:42:59.044318 21638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29f1e7b3165d465c861a7d8d41e76842 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:42:59.044586 21638 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29f1e7b3165d465c861a7d8d41e76842 [term 1 LEADER]: Becoming Leader. State: Replica: 29f1e7b3165d465c861a7d8d41e76842, State: Running, Role: LEADER
I20260927 16:42:59.044937 21638 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29f1e7b3165d465c861a7d8d41e76842 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f1e7b3165d465c861a7d8d41e76842" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37437 } }
I20260927 16:42:59.045312 21628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29f1e7b3165d465c861a7d8d41e76842 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:42:59.046525 21640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f1e7b3165d465c861a7d8d41e76842 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29f1e7b3165d465c861a7d8d41e76842. Latest consensus state: current_term: 1 leader_uuid: "29f1e7b3165d465c861a7d8d41e76842" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f1e7b3165d465c861a7d8d41e76842" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37437 } } }
I20260927 16:42:59.046639 21640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f1e7b3165d465c861a7d8d41e76842 [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:59.047050 21647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:42:59.047015 21639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f1e7b3165d465c861a7d8d41e76842 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29f1e7b3165d465c861a7d8d41e76842" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f1e7b3165d465c861a7d8d41e76842" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37437 } } }
I20260927 16:42:59.047144 21639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f1e7b3165d465c861a7d8d41e76842 [sys.catalog]: This master's current role is: LEADER
I20260927 16:42:59.049757 21647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:42:59.054411 21647 catalog_manager.cc:1347] Generated new cluster ID: dfb6d9e02fda43c5915c37c81aa64ef9
I20260927 16:42:59.054469 21647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:42:59.070798 21647 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:42:59.071099 21647 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:42:59.084410 21647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29f1e7b3165d465c861a7d8d41e76842: Generated new TSK 0
I20260927 16:42:59.085065 21647 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:42:59.095072 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 21560
2026-09-27T16:42:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1347 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:42:59 dist-test-slave-z40b krb5kdc[21659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:42:59 dist-test-slave-z40b krb5kdc[21659](info): set up 2 sockets
Sep 27 16:42:59 dist-test-slave-z40b krb5kdc[21659](info): commencing operation
krb5kdc: starting...
W20260927 16:43:00.103102 392 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:43:00 dist-test-slave-z40b krb5kdc[21659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:43:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:43:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:43:00.214303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43729
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:43:00.316377 21672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:43:00.316620 21672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:43:00.316684 21672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:43:00.320487 21672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:43:00.320554 21672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:43:00.320591 21672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:43:00.320653 21672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:43:00.320689 21672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:43:00.325356 21672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43729
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:43:00.326191 21672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:43:00.327076 21672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:43:00.334566 21680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:43:00.334571 21678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:43:00.334571 21677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:43:00.334908 21672 server_base.cc:1034] running on GCE node
I20260927 16:43:00.335433 21672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:43:00.336373 21672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:43:00.337548 21672 hybrid_clock.cc:648] HybridClock initialized: now 1790527380337531 us; error 28 us; skew 500 ppm
Sep 27 16:43:00 dist-test-slave-z40b krb5kdc[21659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:43:00.340094 21672 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:43:00.340873 21672 webserver.cc:460] Webserver started at http://127.0.0.1:37953/ using document root <none> and password file <none>
I20260927 16:43:00.341353 21672 fs_manager.cc:359] Metadata directory not provided
I20260927 16:43:00.341429 21672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:43:00.341617 21672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:43:00.343532 21672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "b50ccde00bb44ee0b619a62d2bd0fbfb"
format_stamp: "Formatted at 2026-09-27 16:43:00 on dist-test-slave-z40b"
server_key: "b011c291c63eda2f0633cbdb94db0a6a"
server_key_iv: "5e55d9d6ab0cca7f7e300951b7f1b959"
server_key_version: "encryptionkey@0"
I20260927 16:43:00.344082 21672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "b50ccde00bb44ee0b619a62d2bd0fbfb"
format_stamp: "Formatted at 2026-09-27 16:43:00 on dist-test-slave-z40b"
server_key: "b011c291c63eda2f0633cbdb94db0a6a"
server_key_iv: "5e55d9d6ab0cca7f7e300951b7f1b959"
server_key_version: "encryptionkey@0"
I20260927 16:43:00.347389 21672 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:43:00.349531 21687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:43:00.350673 21672 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:43:00.350797 21672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "b50ccde00bb44ee0b619a62d2bd0fbfb"
format_stamp: "Formatted at 2026-09-27 16:43:00 on dist-test-slave-z40b"
server_key: "b011c291c63eda2f0633cbdb94db0a6a"
server_key_iv: "5e55d9d6ab0cca7f7e300951b7f1b959"
server_key_version: "encryptionkey@0"
I20260927 16:43:00.350907 21672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:43:00.366227 21672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:43:00.369155 21672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:43:00.369339 21672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:43:00.375810 21672 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43729
I20260927 16:43:00.375834 21739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43729 every 8 connection(s)
I20260927 16:43:00.376621 21672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:43:00.378996 21740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:43:00.379395 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 21672
I20260927 16:43:00.379541 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:43:00.379792 392 external_mini_cluster.cc:1428] Setting key 9a3be8bbec14f0052c19e1f1bef12040
I20260927 16:43:00.385342 21740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b50ccde00bb44ee0b619a62d2bd0fbfb: Bootstrap starting.
I20260927 16:43:00.387898 21740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b50ccde00bb44ee0b619a62d2bd0fbfb: Neither blocks nor log segments found. Creating new log.
Sep 27 16:43:00 dist-test-slave-z40b krb5kdc[21659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 16:43:00.388628 21740 log.cc:824] T 00000000000000000000000000000000 P b50ccde00bb44ee0b619a62d2bd0fbfb: Log is configured to *not* fsync() on all Append() calls
I20260927 16:43:00.390793 21740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b50ccde00bb44ee0b619a62d2bd0fbfb: No bootstrap required, opened a new log
I20260927 16:43:00.393875 21743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:00.381018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:38363 (local address 127.0.0.1:43729)
0927 16:43:00.381389 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:00.381402 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:00.381434 (+ 32us) server_negotiation.cc:407] Connection header received
0927 16:43:00.382060 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:00.382075 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:00.382397 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:00.382749 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:00.383613 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:00.384560 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:00.385678 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:00.386080 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:00.386182 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:00.388927 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:00.388935 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:00.388952 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:00.388984 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:00.390794 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:00.391304 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:00.391310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:00.391315 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:00.391450 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:00.391767 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:00.391770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:00.391771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:00.392203 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:00.392415 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:00.392640 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 16:43:00.392882 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":105,"threads_started":1}
I20260927 16:43:00.395089 21740 raft_consensus.cc:348] T 00000000000000000000000000000000 P b50ccde00bb44ee0b619a62d2bd0fbfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50ccde00bb44ee0b619a62d2bd0fbfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43729 } }
I20260927 16:43:00.395344 21740 raft_consensus.cc:374] T 00000000000000000000000000000000 P b50ccde00bb44ee0b619a62d2bd0fbfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:43:00.395421 21740 raft_consensus.cc:729] T 00000000000000000000000000000000 P b50ccde00bb44ee0b619a62d2bd0fbfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b50ccde00bb44ee0b619a62d2bd0fbfb, State: Initialized, Role: FOLLOWER
I20260927 16:43:00.395871 21740 consensus_queue.cc:260] T 00000000000000000000000000000000 P b50ccde00bb44ee0b619a62d2bd0fbfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50ccde00bb44ee0b619a62d2bd0fbfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43729 } }
I20260927 16:43:00.396035 21740 raft_consensus.cc:388] T 00000000000000000000000000000000 P b50ccde00bb44ee0b619a62d2bd0fbfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:43:00.396100 21740 raft_consensus.cc:482] T 00000000000000000000000000000000 P b50ccde00bb44ee0b619a62d2bd0fbfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:43:00.396199 21740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b50ccde00bb44ee0b619a62d2bd0fbfb [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:43:00.396690 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:43:00.397049 21740 raft_consensus.cc:504] T 00000000000000000000000000000000 P b50ccde00bb44ee0b619a62d2bd0fbfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50ccde00bb44ee0b619a62d2bd0fbfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43729 } }
I20260927 16:43:00.397368 21740 leader_election.cc:302] T 00000000000000000000000000000000 P b50ccde00bb44ee0b619a62d2bd0fbfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b50ccde00bb44ee0b619a62d2bd0fbfb; no voters:
I20260927 16:43:00.397688 21740 leader_election.cc:288] T 00000000000000000000000000000000 P b50ccde00bb44ee0b619a62d2bd0fbfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:43:00.397791 21746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b50ccde00bb44ee0b619a62d2bd0fbfb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:43:00.398030 21746 raft_consensus.cc:686] T 00000000000000000000000000000000 P b50ccde00bb44ee0b619a62d2bd0fbfb [term 1 LEADER]: Becoming Leader. State: Replica: b50ccde00bb44ee0b619a62d2bd0fbfb, State: Running, Role: LEADER
I20260927 16:43:00.398408 21746 consensus_queue.cc:237] T 00000000000000000000000000000000 P b50ccde00bb44ee0b619a62d2bd0fbfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50ccde00bb44ee0b619a62d2bd0fbfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43729 } }
I20260927 16:43:00.398911 21740 sys_catalog.cc:564] T 00000000000000000000000000000000 P b50ccde00bb44ee0b619a62d2bd0fbfb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:43:00.399972 21750 sys_catalog.cc:455] T 00000000000000000000000000000000 P b50ccde00bb44ee0b619a62d2bd0fbfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b50ccde00bb44ee0b619a62d2bd0fbfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50ccde00bb44ee0b619a62d2bd0fbfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43729 } } }
I20260927 16:43:00.400146 21750 sys_catalog.cc:458] T 00000000000000000000000000000000 P b50ccde00bb44ee0b619a62d2bd0fbfb [sys.catalog]: This master's current role is: LEADER
I20260927 16:43:00.400390 21750 sys_catalog.cc:455] T 00000000000000000000000000000000 P b50ccde00bb44ee0b619a62d2bd0fbfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b50ccde00bb44ee0b619a62d2bd0fbfb. Latest consensus state: current_term: 1 leader_uuid: "b50ccde00bb44ee0b619a62d2bd0fbfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50ccde00bb44ee0b619a62d2bd0fbfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43729 } } }
I20260927 16:43:00.400508 21750 sys_catalog.cc:458] T 00000000000000000000000000000000 P b50ccde00bb44ee0b619a62d2bd0fbfb [sys.catalog]: This master's current role is: LEADER
W20260927 16:43:00.403996 21765 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b50ccde00bb44ee0b619a62d2bd0fbfb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:43:00.404114 21765 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:43:00.404230 21759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:43:00.405021 21759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:43:00.407387 21743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:00.398907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:55159 (local address 127.0.0.1:43729)
0927 16:43:00.399011 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:00.399014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:00.399027 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:43:00.399058 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:00.399061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:00.399098 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:00.399174 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:00.400035 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:00.400712 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:00.401771 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:00.402024 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:00.402110 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:00.403662 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:00.403666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:00.403669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:00.403692 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:00.405298 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:00.405876 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:00.405881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:00.405884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:00.405963 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:00.406344 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:00.406347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:00.406349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:00.406552 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:00.406664 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:00.407122 (+ 458us) server_negotiation.cc:299] Negotiation successful
0927 16:43:00.407249 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 16:43:00.409724 21759 catalog_manager.cc:1347] Generated new cluster ID: 3e1e7b7bd4944607a602ab7335712014
I20260927 16:43:00.409780 21759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:43:00.421262 21759 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:43:00.423038 21759 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:43:00.434000 21759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b50ccde00bb44ee0b619a62d2bd0fbfb: Generated new TSK 0
I20260927 16:43:00.434650 21759 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:43:00.445561 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 21672
W20260927 16:43:00.450650 21657 connection.cc:535] client connection to 127.0.0.1:43729 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43729): Connection reset by peer (error 104)
2026-09-27T16:43:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1349 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:43:00 dist-test-slave-z40b krb5kdc[21771](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:43:00 dist-test-slave-z40b krb5kdc[21771](info): set up 2 sockets
Sep 27 16:43:00 dist-test-slave-z40b krb5kdc[21771](info): commencing operation
krb5kdc: starting...
W20260927 16:43:01.447266 392 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:43:01 dist-test-slave-z40b krb5kdc[21771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:43:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:43:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:43:01.562568 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33793
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:43:01.666234 21784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:43:01.666455 21784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:43:01.666502 21784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:43:01.670274 21784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:43:01.670342 21784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:43:01.670361 21784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:43:01.670377 21784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:43:01.670390 21784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:43:01.674955 21784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33793
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:43:01.675774 21784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:43:01.676563 21784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:43:01.684288 21790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:43:01.684319 21789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:43:01.684460 21784 server_base.cc:1034] running on GCE node
W20260927 16:43:01.684338 21792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:43:01.685109 21784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:43:01.685992 21784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:43:01.687177 21784 hybrid_clock.cc:648] HybridClock initialized: now 1790527381687161 us; error 28 us; skew 500 ppm
Sep 27 16:43:01 dist-test-slave-z40b krb5kdc[21771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:43:01.689698 21784 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:43:01.690450 21784 webserver.cc:460] Webserver started at http://127.0.0.1:41229/ using document root <none> and password file <none>
I20260927 16:43:01.690940 21784 fs_manager.cc:359] Metadata directory not provided
I20260927 16:43:01.691022 21784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:43:01.691285 21784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:43:01.692996 21784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "4f7fd11b99944b61983aa32d727923f0"
format_stamp: "Formatted at 2026-09-27 16:43:01 on dist-test-slave-z40b"
server_key: "fd373634407317c1601a97d46c519d12"
server_key_iv: "d538e80068c70a6dd922fded71d148db"
server_key_version: "encryptionkey@0"
I20260927 16:43:01.693485 21784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "4f7fd11b99944b61983aa32d727923f0"
format_stamp: "Formatted at 2026-09-27 16:43:01 on dist-test-slave-z40b"
server_key: "fd373634407317c1601a97d46c519d12"
server_key_iv: "d538e80068c70a6dd922fded71d148db"
server_key_version: "encryptionkey@0"
I20260927 16:43:01.696605 21784 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260927 16:43:01.698751 21799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:43:01.699817 21784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 16:43:01.699939 21784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "4f7fd11b99944b61983aa32d727923f0"
format_stamp: "Formatted at 2026-09-27 16:43:01 on dist-test-slave-z40b"
server_key: "fd373634407317c1601a97d46c519d12"
server_key_iv: "d538e80068c70a6dd922fded71d148db"
server_key_version: "encryptionkey@0"
I20260927 16:43:01.700032 21784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:43:01.713495 21784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:43:01.716095 21784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:43:01.716280 21784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:43:01.722456 21784 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33793
I20260927 16:43:01.722468 21851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33793 every 8 connection(s)
I20260927 16:43:01.723335 21784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:43:01.726048 21852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:43:01.727674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 21784
I20260927 16:43:01.727806 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:43:01.728081 392 external_mini_cluster.cc:1428] Setting key d71d1c1e6a593deb4a30bdfe467bb738
I20260927 16:43:01.732402 21852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f7fd11b99944b61983aa32d727923f0: Bootstrap starting.
I20260927 16:43:01.734786 21852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f7fd11b99944b61983aa32d727923f0: Neither blocks nor log segments found. Creating new log.
I20260927 16:43:01.735606 21852 log.cc:824] T 00000000000000000000000000000000 P 4f7fd11b99944b61983aa32d727923f0: Log is configured to *not* fsync() on all Append() calls
I20260927 16:43:01.737203 21855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:01.729343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:39767 (local address 127.0.0.1:33793)
0927 16:43:01.729727 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:01.729744 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:01.729782 (+ 38us) server_negotiation.cc:407] Connection header received
0927 16:43:01.730622 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:01.730645 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:01.731037 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:01.731405 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:01.731767 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:01.733090 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:01.733960 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:01.734625 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:01.734788 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:01.735585 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:01.735591 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:01.735606 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:43:01.736059 (+ 453us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:01.736124 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:43:01.736145 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:01.736281 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:01.736459 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 16:43:01.736581 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":97,"threads_started":1}
I20260927 16:43:01.737550 21852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f7fd11b99944b61983aa32d727923f0: No bootstrap required, opened a new log
I20260927 16:43:01.739919 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:43:01.741425 21852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f7fd11b99944b61983aa32d727923f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7fd11b99944b61983aa32d727923f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33793 } }
I20260927 16:43:01.741655 21852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f7fd11b99944b61983aa32d727923f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:43:01.741735 21852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f7fd11b99944b61983aa32d727923f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7fd11b99944b61983aa32d727923f0, State: Initialized, Role: FOLLOWER
I20260927 16:43:01.742238 21852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f7fd11b99944b61983aa32d727923f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7fd11b99944b61983aa32d727923f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33793 } }
I20260927 16:43:01.742398 21852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f7fd11b99944b61983aa32d727923f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:43:01.742467 21852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f7fd11b99944b61983aa32d727923f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:43:01.742563 21852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f7fd11b99944b61983aa32d727923f0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:43:01.743566 21852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f7fd11b99944b61983aa32d727923f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7fd11b99944b61983aa32d727923f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33793 } }
I20260927 16:43:01.743886 21852 leader_election.cc:302] T 00000000000000000000000000000000 P 4f7fd11b99944b61983aa32d727923f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f7fd11b99944b61983aa32d727923f0; no voters:
I20260927 16:43:01.744195 21852 leader_election.cc:288] T 00000000000000000000000000000000 P 4f7fd11b99944b61983aa32d727923f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:43:01.744334 21862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f7fd11b99944b61983aa32d727923f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:43:01.744550 21862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f7fd11b99944b61983aa32d727923f0 [term 1 LEADER]: Becoming Leader. State: Replica: 4f7fd11b99944b61983aa32d727923f0, State: Running, Role: LEADER
I20260927 16:43:01.744875 21862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f7fd11b99944b61983aa32d727923f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7fd11b99944b61983aa32d727923f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33793 } }
I20260927 16:43:01.745471 21852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f7fd11b99944b61983aa32d727923f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:43:01.746080 21855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:01.741758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:58279 (local address 127.0.0.1:33793)
0927 16:43:01.741877 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:01.741881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:01.742085 (+ 204us) server_negotiation.cc:407] Connection header received
0927 16:43:01.742182 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:01.742184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:01.742226 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:01.742299 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:01.742579 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:01.743321 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:01.744181 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:01.744543 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:01.744652 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:01.745588 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:01.745612 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:01.745615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:43:01.745627 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:01.745653 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:43:01.745664 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:01.745741 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:01.745914 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 16:43:01.745945 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:43:01.746744 21863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7fd11b99944b61983aa32d727923f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f7fd11b99944b61983aa32d727923f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7fd11b99944b61983aa32d727923f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33793 } } }
I20260927 16:43:01.746869 21863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7fd11b99944b61983aa32d727923f0 [sys.catalog]: This master's current role is: LEADER
I20260927 16:43:01.747190 21864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7fd11b99944b61983aa32d727923f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f7fd11b99944b61983aa32d727923f0. Latest consensus state: current_term: 1 leader_uuid: "4f7fd11b99944b61983aa32d727923f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7fd11b99944b61983aa32d727923f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33793 } } }
I20260927 16:43:01.747301 21871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:43:01.747365 21864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7fd11b99944b61983aa32d727923f0 [sys.catalog]: This master's current role is: LEADER
I20260927 16:43:01.750051 21871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:43:01.755571 21871 catalog_manager.cc:1347] Generated new cluster ID: 3a7f1781592d4e4780dc88f025008958
I20260927 16:43:01.755653 21871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:43:01.773164 21871 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:43:01.774049 21871 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:43:01.780650 21871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f7fd11b99944b61983aa32d727923f0: Generated new TSK 0
I20260927 16:43:01.781217 21871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:43:01.789522 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 21784
W20260927 16:43:01.794669 21769 connection.cc:535] client connection to 127.0.0.1:33793 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33793): Connection reset by peer (error 104)
2026-09-27T16:43:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:43:01 dist-test-slave-z40b krb5kdc[21883](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:43:01 dist-test-slave-z40b krb5kdc[21883](info): set up 2 sockets
Sep 27 16:43:01 dist-test-slave-z40b krb5kdc[21883](info): commencing operation
krb5kdc: starting...
W20260927 16:43:02.795281 392 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:43:02 dist-test-slave-z40b krb5kdc[21883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:43:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:43:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:43:02.913717 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38729
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:43:03.016330 21896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:43:03.016556 21896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:43:03.016593 21896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:43:03.020445 21896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:43:03.020495 21896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:43:03.020514 21896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:43:03.020529 21896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:43:03.020545 21896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:43:03.025383 21896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38729
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790527121515808-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:43:03.026255 21896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:43:03.027208 21896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:43:03.034845 21901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:43:03.034849 21902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:43:03.034894 21896 server_base.cc:1034] running on GCE node
W20260927 16:43:03.034837 21904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:43:03.035641 21896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:43:03.036571 21896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:43:03.037753 21896 hybrid_clock.cc:648] HybridClock initialized: now 1790527383037729 us; error 44 us; skew 500 ppm
Sep 27 16:43:03 dist-test-slave-z40b krb5kdc[21883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:43:03.044039 21896 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:43:03.044879 21896 webserver.cc:460] Webserver started at http://127.0.0.1:35593/ using document root <none> and password file <none>
I20260927 16:43:03.045410 21896 fs_manager.cc:359] Metadata directory not provided
I20260927 16:43:03.045468 21896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:43:03.045692 21896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:43:03.047691 21896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "e11cc717b6244aa9b3a05465df36c48e"
format_stamp: "Formatted at 2026-09-27 16:43:03 on dist-test-slave-z40b"
server_key: "b1b5dfa394b916fa48916851f9bb8337"
server_key_iv: "334635ff49e78e4964ae235681a63010"
server_key_version: "encryptionkey@0"
I20260927 16:43:03.048233 21896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "e11cc717b6244aa9b3a05465df36c48e"
format_stamp: "Formatted at 2026-09-27 16:43:03 on dist-test-slave-z40b"
server_key: "b1b5dfa394b916fa48916851f9bb8337"
server_key_iv: "334635ff49e78e4964ae235681a63010"
server_key_version: "encryptionkey@0"
I20260927 16:43:03.051604 21896 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:43:03.053944 21911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:43:03.055020 21896 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:43:03.055140 21896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "e11cc717b6244aa9b3a05465df36c48e"
format_stamp: "Formatted at 2026-09-27 16:43:03 on dist-test-slave-z40b"
server_key: "b1b5dfa394b916fa48916851f9bb8337"
server_key_iv: "334635ff49e78e4964ae235681a63010"
server_key_version: "encryptionkey@0"
I20260927 16:43:03.055296 21896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:43:03.059512 21896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:43:03.062042 21896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:43:03.062217 21896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:43:03.068701 21896 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38729
I20260927 16:43:03.068723 21963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38729 every 8 connection(s)
I20260927 16:43:03.069526 21896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:43:03.071949 21964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:43:03.076817 21964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e11cc717b6244aa9b3a05465df36c48e: Bootstrap starting.
I20260927 16:43:03.078876 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 21896
I20260927 16:43:03.079010 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:43:03.079154 21964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e11cc717b6244aa9b3a05465df36c48e: Neither blocks nor log segments found. Creating new log.
I20260927 16:43:03.079334 392 external_mini_cluster.cc:1428] Setting key 9b9ff589be933cd062bb427bd391a91d
I20260927 16:43:03.080035 21964 log.cc:824] T 00000000000000000000000000000000 P e11cc717b6244aa9b3a05465df36c48e: Log is configured to *not* fsync() on all Append() calls
I20260927 16:43:03.081789 21964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e11cc717b6244aa9b3a05465df36c48e: No bootstrap required, opened a new log
I20260927 16:43:03.083840 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:03.080360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:36005 (local address 127.0.0.1:38729)
0927 16:43:03.080769 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:03.080785 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:03.080832 (+ 47us) server_negotiation.cc:407] Connection header received
0927 16:43:03.081387 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:03.081418 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:03.081667 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:03.081982 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:03.082718 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:03.082723 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:03.082737 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:43:03.082754 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:03.082800 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:43:03.082823 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:03.082934 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:03.083031 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 16:43:03.083168 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":113,"threads_started":1}
I20260927 16:43:03.085705 21964 raft_consensus.cc:348] T 00000000000000000000000000000000 P e11cc717b6244aa9b3a05465df36c48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11cc717b6244aa9b3a05465df36c48e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38729 } }
I20260927 16:43:03.085937 21964 raft_consensus.cc:374] T 00000000000000000000000000000000 P e11cc717b6244aa9b3a05465df36c48e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:43:03.085994 21964 raft_consensus.cc:729] T 00000000000000000000000000000000 P e11cc717b6244aa9b3a05465df36c48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e11cc717b6244aa9b3a05465df36c48e, State: Initialized, Role: FOLLOWER
I20260927 16:43:03.086436 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:43:03.086452 21964 consensus_queue.cc:260] T 00000000000000000000000000000000 P e11cc717b6244aa9b3a05465df36c48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11cc717b6244aa9b3a05465df36c48e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38729 } }
I20260927 16:43:03.086613 21964 raft_consensus.cc:388] T 00000000000000000000000000000000 P e11cc717b6244aa9b3a05465df36c48e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:43:03.086683 21964 raft_consensus.cc:482] T 00000000000000000000000000000000 P e11cc717b6244aa9b3a05465df36c48e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:43:03.086803 21964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e11cc717b6244aa9b3a05465df36c48e [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:43:03.087694 21964 raft_consensus.cc:504] T 00000000000000000000000000000000 P e11cc717b6244aa9b3a05465df36c48e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11cc717b6244aa9b3a05465df36c48e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38729 } }
I20260927 16:43:03.088071 21964 leader_election.cc:302] T 00000000000000000000000000000000 P e11cc717b6244aa9b3a05465df36c48e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e11cc717b6244aa9b3a05465df36c48e; no voters:
I20260927 16:43:03.088382 21964 leader_election.cc:288] T 00000000000000000000000000000000 P e11cc717b6244aa9b3a05465df36c48e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:43:03.088430 21973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e11cc717b6244aa9b3a05465df36c48e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:43:03.088774 21973 raft_consensus.cc:686] T 00000000000000000000000000000000 P e11cc717b6244aa9b3a05465df36c48e [term 1 LEADER]: Becoming Leader. State: Replica: e11cc717b6244aa9b3a05465df36c48e, State: Running, Role: LEADER
I20260927 16:43:03.089075 21973 consensus_queue.cc:237] T 00000000000000000000000000000000 P e11cc717b6244aa9b3a05465df36c48e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11cc717b6244aa9b3a05465df36c48e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38729 } }
I20260927 16:43:03.089649 21964 sys_catalog.cc:564] T 00000000000000000000000000000000 P e11cc717b6244aa9b3a05465df36c48e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:43:03.090642 21974 sys_catalog.cc:455] T 00000000000000000000000000000000 P e11cc717b6244aa9b3a05465df36c48e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e11cc717b6244aa9b3a05465df36c48e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11cc717b6244aa9b3a05465df36c48e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38729 } } }
I20260927 16:43:03.090754 21974 sys_catalog.cc:458] T 00000000000000000000000000000000 P e11cc717b6244aa9b3a05465df36c48e [sys.catalog]: This master's current role is: LEADER
I20260927 16:43:03.090900 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:03.088872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:59661 (local address 127.0.0.1:38729)
0927 16:43:03.088958 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:03.088962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:03.089311 (+ 349us) server_negotiation.cc:407] Connection header received
0927 16:43:03.089447 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:03.089450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:03.089482 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:03.089559 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:03.090371 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:03.090375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:03.090378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:43:03.090391 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:03.090430 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:43:03.090446 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:03.090510 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:03.090706 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 16:43:03.090744 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 16:43:03.091209 21976 sys_catalog.cc:455] T 00000000000000000000000000000000 P e11cc717b6244aa9b3a05465df36c48e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e11cc717b6244aa9b3a05465df36c48e. Latest consensus state: current_term: 1 leader_uuid: "e11cc717b6244aa9b3a05465df36c48e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11cc717b6244aa9b3a05465df36c48e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38729 } } }
I20260927 16:43:03.091382 21976 sys_catalog.cc:458] T 00000000000000000000000000000000 P e11cc717b6244aa9b3a05465df36c48e [sys.catalog]: This master's current role is: LEADER
I20260927 16:43:03.091670 21983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:43:03.094230 21983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:43:03.099170 21983 catalog_manager.cc:1347] Generated new cluster ID: 35241e66df3845eeac5a88f1f766335c
I20260927 16:43:03.099255 21983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:43:03.112300 21983 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:43:03.112596 21983 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:43:03.122942 21983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e11cc717b6244aa9b3a05465df36c48e: Generated new TSK 0
I20260927 16:43:03.123590 21983 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:43:03.133880 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 21896
2026-09-27T16:43:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:43:03 dist-test-slave-z40b krb5kdc[21995](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:43:03 dist-test-slave-z40b krb5kdc[21995](info): set up 2 sockets
Sep 27 16:43:03 dist-test-slave-z40b krb5kdc[21995](info): commencing operation
krb5kdc: starting...
W20260927 16:43:04.135362 392 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:43:04 dist-test-slave-z40b krb5kdc[21995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:43:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:43:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:43:04.248100 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34085
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:43:04.348698 22008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:43:04.348923 22008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:43:04.348960 22008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:43:04.352514 22008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:43:04.352555 22008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:43:04.352584 22008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:43:04.352598 22008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:43:04.352631 22008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:43:04.357163 22008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34085
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:43:04.357995 22008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:43:04.358829 22008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:43:04.366415 22014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:43:04.366453 22013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:43:04.366453 22016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:43:04.366484 22008 server_base.cc:1034] running on GCE node
I20260927 16:43:04.367147 22008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:43:04.368116 22008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:43:04.369271 22008 hybrid_clock.cc:648] HybridClock initialized: now 1790527384369239 us; error 44 us; skew 500 ppm
Sep 27 16:43:04 dist-test-slave-z40b krb5kdc[21995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:43:04.371836 22008 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:43:04.372634 22008 webserver.cc:460] Webserver started at http://127.0.98.62:34543/ using document root <none> and password file <none>
I20260927 16:43:04.373119 22008 fs_manager.cc:359] Metadata directory not provided
I20260927 16:43:04.373203 22008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:43:04.373417 22008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:43:04.375276 22008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "87b1f0dde5cf44daac554c0a08ec292a"
format_stamp: "Formatted at 2026-09-27 16:43:04 on dist-test-slave-z40b"
server_key: "529ba232ddbfa2f65c8d55fd0e593822"
server_key_iv: "21704e2e111cefd5169e2e0b517f4bc4"
server_key_version: "encryptionkey@0"
I20260927 16:43:04.375834 22008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "87b1f0dde5cf44daac554c0a08ec292a"
format_stamp: "Formatted at 2026-09-27 16:43:04 on dist-test-slave-z40b"
server_key: "529ba232ddbfa2f65c8d55fd0e593822"
server_key_iv: "21704e2e111cefd5169e2e0b517f4bc4"
server_key_version: "encryptionkey@0"
I20260927 16:43:04.380055 22008 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 16:43:04.382265 22023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:43:04.383625 22008 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:43:04.383719 22008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "87b1f0dde5cf44daac554c0a08ec292a"
format_stamp: "Formatted at 2026-09-27 16:43:04 on dist-test-slave-z40b"
server_key: "529ba232ddbfa2f65c8d55fd0e593822"
server_key_iv: "21704e2e111cefd5169e2e0b517f4bc4"
server_key_version: "encryptionkey@0"
I20260927 16:43:04.383823 22008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:43:04.395040 22008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:43:04.397538 22008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:43:04.397679 22008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:43:04.403959 22008 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34085
I20260927 16:43:04.404045 22075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34085 every 8 connection(s)
I20260927 16:43:04.404821 22008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:43:04.407395 22076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:43:04.412505 22076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b1f0dde5cf44daac554c0a08ec292a: Bootstrap starting.
I20260927 16:43:04.413317 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 22008
I20260927 16:43:04.413460 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:43:04.413717 392 external_mini_cluster.cc:1428] Setting key 78b18818f79588dc76a77fd724731208
I20260927 16:43:04.415158 22076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87b1f0dde5cf44daac554c0a08ec292a: Neither blocks nor log segments found. Creating new log.
I20260927 16:43:04.416042 22076 log.cc:824] T 00000000000000000000000000000000 P 87b1f0dde5cf44daac554c0a08ec292a: Log is configured to *not* fsync() on all Append() calls
I20260927 16:43:04.418107 22076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b1f0dde5cf44daac554c0a08ec292a: No bootstrap required, opened a new log
I20260927 16:43:04.421929 22076 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87b1f0dde5cf44daac554c0a08ec292a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b1f0dde5cf44daac554c0a08ec292a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34085 } }
I20260927 16:43:04.422119 22076 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87b1f0dde5cf44daac554c0a08ec292a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:43:04.422151 22076 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87b1f0dde5cf44daac554c0a08ec292a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b1f0dde5cf44daac554c0a08ec292a, State: Initialized, Role: FOLLOWER
I20260927 16:43:04.422655 22076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87b1f0dde5cf44daac554c0a08ec292a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b1f0dde5cf44daac554c0a08ec292a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34085 } }
Sep 27 16:43:04 dist-test-slave-z40b krb5kdc[21995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:43:04.422804 22076 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87b1f0dde5cf44daac554c0a08ec292a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:43:04.422894 22076 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87b1f0dde5cf44daac554c0a08ec292a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:43:04.423015 22076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87b1f0dde5cf44daac554c0a08ec292a [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:43:04.424209 22076 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87b1f0dde5cf44daac554c0a08ec292a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b1f0dde5cf44daac554c0a08ec292a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34085 } }
I20260927 16:43:04.424592 22076 leader_election.cc:302] T 00000000000000000000000000000000 P 87b1f0dde5cf44daac554c0a08ec292a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87b1f0dde5cf44daac554c0a08ec292a; no voters:
I20260927 16:43:04.424922 22076 leader_election.cc:288] T 00000000000000000000000000000000 P 87b1f0dde5cf44daac554c0a08ec292a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:43:04.425016 22081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87b1f0dde5cf44daac554c0a08ec292a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:43:04.425238 22081 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87b1f0dde5cf44daac554c0a08ec292a [term 1 LEADER]: Becoming Leader. State: Replica: 87b1f0dde5cf44daac554c0a08ec292a, State: Running, Role: LEADER
I20260927 16:43:04.425571 22081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87b1f0dde5cf44daac554c0a08ec292a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b1f0dde5cf44daac554c0a08ec292a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34085 } }
I20260927 16:43:04.426142 22076 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87b1f0dde5cf44daac554c0a08ec292a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:43:04.427301 22082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b1f0dde5cf44daac554c0a08ec292a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87b1f0dde5cf44daac554c0a08ec292a. Latest consensus state: current_term: 1 leader_uuid: "87b1f0dde5cf44daac554c0a08ec292a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b1f0dde5cf44daac554c0a08ec292a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34085 } } }
I20260927 16:43:04.427631 22082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b1f0dde5cf44daac554c0a08ec292a [sys.catalog]: This master's current role is: LEADER
I20260927 16:43:04.427531 22083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b1f0dde5cf44daac554c0a08ec292a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87b1f0dde5cf44daac554c0a08ec292a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b1f0dde5cf44daac554c0a08ec292a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34085 } } }
I20260927 16:43:04.428071 22083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b1f0dde5cf44daac554c0a08ec292a [sys.catalog]: This master's current role is: LEADER
I20260927 16:43:04.428812 22079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:04.414770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35279 (local address 127.0.98.62:34085)
0927 16:43:04.415287 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:04.415307 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:04.415349 (+ 42us) server_negotiation.cc:407] Connection header received
0927 16:43:04.416192 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:04.416214 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:04.416612 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:04.416965 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:04.417645 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:04.419100 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:04.419995 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:04.420723 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:04.420924 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:04.423509 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:04.423516 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:04.423531 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:04.423560 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:04.425514 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:04.426054 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:04.426059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:04.426064 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:04.426197 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:04.426594 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:04.426598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:04.426600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:04.427151 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:04.427443 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:04.427753 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 16:43:04.427978 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":117,"threads_started":1}
W20260927 16:43:04.430879 22096 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 87b1f0dde5cf44daac554c0a08ec292a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:43:04.431022 22096 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:43:04.431141 22095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:43:04.431716 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:43:04.432055 22095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:43:04.437875 22095 catalog_manager.cc:1347] Generated new cluster ID: 1d9aaa46c98544d4bcb921fda606a180
I20260927 16:43:04.437979 22095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:43:04.443370 22079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:04.435341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52485 (local address 127.0.98.62:34085)
0927 16:43:04.435441 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:04.435445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:04.435457 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:43:04.435493 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:04.435495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:04.435525 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:04.435608 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:04.436346 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:04.437449 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:04.438470 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:04.438738 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:04.438840 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:04.440184 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:04.440188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:04.440190 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:04.440212 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:04.441613 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:04.442072 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:04.442075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:04.442076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:04.442146 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:04.442479 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:04.442482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:04.442484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:04.442645 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:04.442808 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:04.443062 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 16:43:04.443147 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 16:43:04.451431 22095 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:43:04.452610 22095 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:43:04.457825 22095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87b1f0dde5cf44daac554c0a08ec292a: Generated new TSK 0
I20260927 16:43:04.458556 22095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:43:04.469700 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 22008
W20260927 16:43:04.474671 21993 connection.cc:535] client connection to 127.0.98.62:34085 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:34085): Connection reset by peer (error 104)
2026-09-27T16:43:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:43:04 dist-test-slave-z40b krb5kdc[22107](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:43:04 dist-test-slave-z40b krb5kdc[22107](info): set up 2 sockets
Sep 27 16:43:04 dist-test-slave-z40b krb5kdc[22107](info): commencing operation
krb5kdc: starting...
W20260927 16:43:05.483072 392 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:43:05 dist-test-slave-z40b krb5kdc[22107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:43:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:43:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:43:05.596038 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43043
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:43:05.703437 22120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:43:05.703675 22120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:43:05.703711 22120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:43:05.707691 22120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:43:05.707743 22120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:43:05.707762 22120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:43:05.707834 22120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:43:05.707892 22120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:43:05.712819 22120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43043
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:43:05.713785 22120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:43:05.714747 22120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:43:05.722817 22125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:43:05.722808 22126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:43:05.722931 22120 server_base.cc:1034] running on GCE node
W20260927 16:43:05.723096 22128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:43:05.723618 22120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:43:05.724545 22120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:43:05.725699 22120 hybrid_clock.cc:648] HybridClock initialized: now 1790527385725681 us; error 28 us; skew 500 ppm
Sep 27 16:43:05 dist-test-slave-z40b krb5kdc[22107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:43:05.728335 22120 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:43:05.729143 22120 webserver.cc:460] Webserver started at http://127.0.98.62:39647/ using document root <none> and password file <none>
I20260927 16:43:05.729657 22120 fs_manager.cc:359] Metadata directory not provided
I20260927 16:43:05.729712 22120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:43:05.729947 22120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:43:05.731869 22120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "0131ebbe93e140799ff05c5ac1c04c56"
format_stamp: "Formatted at 2026-09-27 16:43:05 on dist-test-slave-z40b"
server_key: "ade9ccf7e0349dece1332bbaed03eab4"
server_key_iv: "008e1104efac2a62df79dd84abdfd754"
server_key_version: "encryptionkey@0"
I20260927 16:43:05.732375 22120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "0131ebbe93e140799ff05c5ac1c04c56"
format_stamp: "Formatted at 2026-09-27 16:43:05 on dist-test-slave-z40b"
server_key: "ade9ccf7e0349dece1332bbaed03eab4"
server_key_iv: "008e1104efac2a62df79dd84abdfd754"
server_key_version: "encryptionkey@0"
I20260927 16:43:05.735839 22120 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:43:05.738125 22135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:43:05.739302 22120 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:43:05.739423 22120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "0131ebbe93e140799ff05c5ac1c04c56"
format_stamp: "Formatted at 2026-09-27 16:43:05 on dist-test-slave-z40b"
server_key: "ade9ccf7e0349dece1332bbaed03eab4"
server_key_iv: "008e1104efac2a62df79dd84abdfd754"
server_key_version: "encryptionkey@0"
I20260927 16:43:05.739522 22120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:43:05.761029 22120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:43:05.763742 22120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:43:05.763934 22120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:43:05.770568 22120 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43043
I20260927 16:43:05.770586 22187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43043 every 8 connection(s)
I20260927 16:43:05.771471 22120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:43:05.774151 22188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:43:05.779320 22188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0131ebbe93e140799ff05c5ac1c04c56: Bootstrap starting.
I20260927 16:43:05.781606 22188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0131ebbe93e140799ff05c5ac1c04c56: Neither blocks nor log segments found. Creating new log.
I20260927 16:43:05.781704 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 22120
I20260927 16:43:05.781865 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:43:05.782091 392 external_mini_cluster.cc:1428] Setting key 87c3e6ddca1eb7c6cb190190c729c09e
I20260927 16:43:05.782413 22188 log.cc:824] T 00000000000000000000000000000000 P 0131ebbe93e140799ff05c5ac1c04c56: Log is configured to *not* fsync() on all Append() calls
I20260927 16:43:05.784247 22188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0131ebbe93e140799ff05c5ac1c04c56: No bootstrap required, opened a new log
I20260927 16:43:05.788178 22188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0131ebbe93e140799ff05c5ac1c04c56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0131ebbe93e140799ff05c5ac1c04c56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43043 } }
I20260927 16:43:05.788448 22188 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0131ebbe93e140799ff05c5ac1c04c56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:43:05.788509 22188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0131ebbe93e140799ff05c5ac1c04c56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0131ebbe93e140799ff05c5ac1c04c56, State: Initialized, Role: FOLLOWER
I20260927 16:43:05.788981 22188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0131ebbe93e140799ff05c5ac1c04c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0131ebbe93e140799ff05c5ac1c04c56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43043 } }
I20260927 16:43:05.789100 22188 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0131ebbe93e140799ff05c5ac1c04c56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:43:05.789214 22188 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0131ebbe93e140799ff05c5ac1c04c56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:43:05.789310 22188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0131ebbe93e140799ff05c5ac1c04c56 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:43:05.790221 22188 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0131ebbe93e140799ff05c5ac1c04c56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0131ebbe93e140799ff05c5ac1c04c56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43043 } }
I20260927 16:43:05.790318 22192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:05.783084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40937 (local address 127.0.98.62:43043)
0927 16:43:05.783451 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:05.783464 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:05.783495 (+ 31us) server_negotiation.cc:407] Connection header received
0927 16:43:05.784037 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:05.784051 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:05.784406 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:05.784660 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:05.784992 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:05.785879 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:05.787284 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:05.787720 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:05.787837 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:05.788701 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:05.788706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:05.788737 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:43:05.789134 (+ 397us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:05.789200 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:43:05.789234 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:05.789382 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:05.789547 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 16:43:05.789674 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":125,"threads_started":1}
I20260927 16:43:05.790601 22188 leader_election.cc:302] T 00000000000000000000000000000000 P 0131ebbe93e140799ff05c5ac1c04c56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0131ebbe93e140799ff05c5ac1c04c56; no voters:
I20260927 16:43:05.790904 22188 leader_election.cc:288] T 00000000000000000000000000000000 P 0131ebbe93e140799ff05c5ac1c04c56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:43:05.790977 22193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0131ebbe93e140799ff05c5ac1c04c56 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:43:05.791184 22193 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0131ebbe93e140799ff05c5ac1c04c56 [term 1 LEADER]: Becoming Leader. State: Replica: 0131ebbe93e140799ff05c5ac1c04c56, State: Running, Role: LEADER
I20260927 16:43:05.791518 22193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0131ebbe93e140799ff05c5ac1c04c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0131ebbe93e140799ff05c5ac1c04c56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43043 } }
I20260927 16:43:05.792042 22188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0131ebbe93e140799ff05c5ac1c04c56 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:43:05.792740 22195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0131ebbe93e140799ff05c5ac1c04c56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0131ebbe93e140799ff05c5ac1c04c56. Latest consensus state: current_term: 1 leader_uuid: "0131ebbe93e140799ff05c5ac1c04c56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0131ebbe93e140799ff05c5ac1c04c56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43043 } } }
I20260927 16:43:05.793411 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:43:05.795503 22195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0131ebbe93e140799ff05c5ac1c04c56 [sys.catalog]: This master's current role is: LEADER
I20260927 16:43:05.795554 22194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0131ebbe93e140799ff05c5ac1c04c56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0131ebbe93e140799ff05c5ac1c04c56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0131ebbe93e140799ff05c5ac1c04c56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43043 } } }
I20260927 16:43:05.796007 22194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0131ebbe93e140799ff05c5ac1c04c56 [sys.catalog]: This master's current role is: LEADER
W20260927 16:43:05.797008 22207 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0131ebbe93e140799ff05c5ac1c04c56: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:43:05.797086 22207 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:43:05.797230 22215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:43:05.797995 22215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:43:05.799978 22192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:05.795546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42107 (local address 127.0.98.62:43043)
0927 16:43:05.795666 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:05.795673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:05.795947 (+ 274us) server_negotiation.cc:407] Connection header received
0927 16:43:05.796064 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:05.796068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:05.796111 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:05.796185 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:05.796446 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:05.796918 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:05.798089 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:05.798388 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:05.798495 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:05.799417 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:05.799421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:05.799423 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:43:05.799436 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:05.799461 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:43:05.799475 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:05.799549 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:05.799807 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 16:43:05.799855 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:43:05.803395 22215 catalog_manager.cc:1347] Generated new cluster ID: 4888328cde6c4ff09479e435878b99df
I20260927 16:43:05.803455 22215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:43:05.818691 22215 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:43:05.819628 22215 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:43:05.828828 22215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0131ebbe93e140799ff05c5ac1c04c56: Generated new TSK 0
I20260927 16:43:05.829457 22215 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:43:05.840839 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 22120
W20260927 16:43:05.845713 22105 connection.cc:535] client connection to 127.0.98.62:43043 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:43043): Connection reset by peer (error 104)
2026-09-27T16:43:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1370 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:43:05 dist-test-slave-z40b krb5kdc[22219](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:43:05 dist-test-slave-z40b krb5kdc[22219](info): set up 2 sockets
Sep 27 16:43:05 dist-test-slave-z40b krb5kdc[22219](info): commencing operation
krb5kdc: starting...
W20260927 16:43:06.851405 392 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:43:06 dist-test-slave-z40b krb5kdc[22219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:43:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:43:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:43:06.964202 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41509
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:43:07.068331 22232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:43:07.068580 22232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:43:07.068620 22232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:43:07.072633 22232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:43:07.072685 22232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:43:07.072705 22232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:43:07.072754 22232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:43:07.072786 22232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:43:07.077792 22232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41509
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:43:07.078800 22232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:43:07.079787 22232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:43:07.087692 22237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:43:07.087680 22240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:43:07.087682 22238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:43:07.087841 22232 server_base.cc:1034] running on GCE node
I20260927 16:43:07.088588 22232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:43:07.089572 22232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:43:07.090772 22232 hybrid_clock.cc:648] HybridClock initialized: now 1790527387090760 us; error 32 us; skew 500 ppm
Sep 27 16:43:07 dist-test-slave-z40b krb5kdc[22219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:43:07.093333 22232 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:43:07.094164 22232 webserver.cc:460] Webserver started at http://127.0.98.62:42209/ using document root <none> and password file <none>
I20260927 16:43:07.094667 22232 fs_manager.cc:359] Metadata directory not provided
I20260927 16:43:07.094760 22232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:43:07.094976 22232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:43:07.096966 22232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "95f6a78589994e10a144670478ca93a0"
format_stamp: "Formatted at 2026-09-27 16:43:07 on dist-test-slave-z40b"
server_key: "89dbb4556154e66215adb45ed341cb3e"
server_key_iv: "5eb7592e74573ffbe4ca7e63da6190b6"
server_key_version: "encryptionkey@0"
I20260927 16:43:07.097503 22232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "95f6a78589994e10a144670478ca93a0"
format_stamp: "Formatted at 2026-09-27 16:43:07 on dist-test-slave-z40b"
server_key: "89dbb4556154e66215adb45ed341cb3e"
server_key_iv: "5eb7592e74573ffbe4ca7e63da6190b6"
server_key_version: "encryptionkey@0"
I20260927 16:43:07.100930 22232 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:43:07.103055 22247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:43:07.104206 22232 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:43:07.104336 22232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "95f6a78589994e10a144670478ca93a0"
format_stamp: "Formatted at 2026-09-27 16:43:07 on dist-test-slave-z40b"
server_key: "89dbb4556154e66215adb45ed341cb3e"
server_key_iv: "5eb7592e74573ffbe4ca7e63da6190b6"
server_key_version: "encryptionkey@0"
I20260927 16:43:07.104439 22232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:43:07.108592 22232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:43:07.111691 22232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:43:07.111872 22232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:43:07.118628 22232 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41509
I20260927 16:43:07.118652 22299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41509 every 8 connection(s)
I20260927 16:43:07.119515 22232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:43:07.122007 22300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:43:07.127046 22300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95f6a78589994e10a144670478ca93a0: Bootstrap starting.
I20260927 16:43:07.129218 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 22232
I20260927 16:43:07.129370 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:43:07.129459 22300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95f6a78589994e10a144670478ca93a0: Neither blocks nor log segments found. Creating new log.
I20260927 16:43:07.129685 392 external_mini_cluster.cc:1428] Setting key a3f19e7f4b7ecc483f879e74f96be114
I20260927 16:43:07.130239 22300 log.cc:824] T 00000000000000000000000000000000 P 95f6a78589994e10a144670478ca93a0: Log is configured to *not* fsync() on all Append() calls
I20260927 16:43:07.132156 22300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95f6a78589994e10a144670478ca93a0: No bootstrap required, opened a new log
I20260927 16:43:07.134135 22304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:07.130706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52221 (local address 127.0.98.62:41509)
0927 16:43:07.131065 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:07.131079 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:07.131123 (+ 44us) server_negotiation.cc:407] Connection header received
0927 16:43:07.131775 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:07.131798 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:07.132046 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:07.132370 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:07.133090 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:07.133094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:07.133104 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:43:07.133113 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:07.133158 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:43:07.133175 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:07.133277 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:07.133390 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 16:43:07.133525 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":109,"threads_started":1}
I20260927 16:43:07.135823 22300 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95f6a78589994e10a144670478ca93a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95f6a78589994e10a144670478ca93a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41509 } }
I20260927 16:43:07.136096 22300 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95f6a78589994e10a144670478ca93a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:43:07.136173 22300 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95f6a78589994e10a144670478ca93a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95f6a78589994e10a144670478ca93a0, State: Initialized, Role: FOLLOWER
I20260927 16:43:07.136590 22300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95f6a78589994e10a144670478ca93a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95f6a78589994e10a144670478ca93a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41509 } }
I20260927 16:43:07.136750 22300 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95f6a78589994e10a144670478ca93a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:43:07.136823 22300 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95f6a78589994e10a144670478ca93a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:43:07.136868 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:43:07.136917 22300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95f6a78589994e10a144670478ca93a0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:43:07.137971 22300 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95f6a78589994e10a144670478ca93a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95f6a78589994e10a144670478ca93a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41509 } }
I20260927 16:43:07.138336 22300 leader_election.cc:302] T 00000000000000000000000000000000 P 95f6a78589994e10a144670478ca93a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95f6a78589994e10a144670478ca93a0; no voters:
I20260927 16:43:07.138638 22300 leader_election.cc:288] T 00000000000000000000000000000000 P 95f6a78589994e10a144670478ca93a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:43:07.138736 22309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95f6a78589994e10a144670478ca93a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:43:07.139001 22309 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95f6a78589994e10a144670478ca93a0 [term 1 LEADER]: Becoming Leader. State: Replica: 95f6a78589994e10a144670478ca93a0, State: Running, Role: LEADER
I20260927 16:43:07.139461 22309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95f6a78589994e10a144670478ca93a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95f6a78589994e10a144670478ca93a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41509 } }
I20260927 16:43:07.139943 22300 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95f6a78589994e10a144670478ca93a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:43:07.140893 22304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:07.139055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60477 (local address 127.0.98.62:41509)
0927 16:43:07.139164 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:07.139167 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:07.139399 (+ 232us) server_negotiation.cc:407] Connection header received
0927 16:43:07.139557 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:07.139562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:07.139609 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:07.139683 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:07.140451 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:07.140457 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:07.140459 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:43:07.140474 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:07.140502 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:43:07.140515 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:07.140594 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:07.140756 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 16:43:07.140789 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 16:43:07.141472 22311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95f6a78589994e10a144670478ca93a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95f6a78589994e10a144670478ca93a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95f6a78589994e10a144670478ca93a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41509 } } }
I20260927 16:43:07.141670 22311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95f6a78589994e10a144670478ca93a0 [sys.catalog]: This master's current role is: LEADER
I20260927 16:43:07.142354 22324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:43:07.141981 22310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95f6a78589994e10a144670478ca93a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95f6a78589994e10a144670478ca93a0. Latest consensus state: current_term: 1 leader_uuid: "95f6a78589994e10a144670478ca93a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95f6a78589994e10a144670478ca93a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41509 } } }
I20260927 16:43:07.143272 22310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95f6a78589994e10a144670478ca93a0 [sys.catalog]: This master's current role is: LEADER
I20260927 16:43:07.144994 22324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:43:07.149658 22324 catalog_manager.cc:1347] Generated new cluster ID: 8f6b80d265b44215a82a503b4753f31e
I20260927 16:43:07.149715 22324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:43:07.180469 22324 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:43:07.180785 22324 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:43:07.188755 22324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95f6a78589994e10a144670478ca93a0: Generated new TSK 0
I20260927 16:43:07.189424 22324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:43:07.202371 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 22232
2026-09-27T16:43:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1361 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:43:07 dist-test-slave-z40b krb5kdc[22331](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:43:07 dist-test-slave-z40b krb5kdc[22331](info): set up 2 sockets
Sep 27 16:43:07 dist-test-slave-z40b krb5kdc[22331](info): commencing operation
krb5kdc: starting...
W20260927 16:43:08.211329 392 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:43:08 dist-test-slave-z40b krb5kdc[22331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:43:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:43:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:43:08.324400 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40667
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:43:08.430115 22344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:43:08.430390 22344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:43:08.430456 22344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:43:08.434283 22344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:43:08.434360 22344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:43:08.434396 22344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:43:08.434451 22344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:43:08.434486 22344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:43:08.439791 22344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:43:08.440770 22344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:43:08.441732 22344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:43:08.450167 22344 server_base.cc:1034] running on GCE node
W20260927 16:43:08.450078 22349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:43:08.450095 22352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:43:08.450095 22350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:43:08.450870 22344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:43:08.451953 22344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:43:08.453145 22344 hybrid_clock.cc:648] HybridClock initialized: now 1790527388453117 us; error 39 us; skew 500 ppm
Sep 27 16:43:08 dist-test-slave-z40b krb5kdc[22331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:43:08.458425 22344 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:43:08.459290 22344 webserver.cc:460] Webserver started at http://127.0.98.62:44493/ using document root <none> and password file <none>
I20260927 16:43:08.459926 22344 fs_manager.cc:359] Metadata directory not provided
I20260927 16:43:08.460012 22344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:43:08.460214 22344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:43:08.462119 22344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "678ce655693248d39b959642c5d8b567"
format_stamp: "Formatted at 2026-09-27 16:43:08 on dist-test-slave-z40b"
server_key: "95234c158844849e6da59fb0c8f831a9"
server_key_iv: "a46c996e65c4d2ab90ad9ec7c49dd56d"
server_key_version: "encryptionkey@0"
I20260927 16:43:08.462661 22344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "678ce655693248d39b959642c5d8b567"
format_stamp: "Formatted at 2026-09-27 16:43:08 on dist-test-slave-z40b"
server_key: "95234c158844849e6da59fb0c8f831a9"
server_key_iv: "a46c996e65c4d2ab90ad9ec7c49dd56d"
server_key_version: "encryptionkey@0"
I20260927 16:43:08.466084 22344 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:43:08.468405 22359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:43:08.469532 22344 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:43:08.469683 22344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "678ce655693248d39b959642c5d8b567"
format_stamp: "Formatted at 2026-09-27 16:43:08 on dist-test-slave-z40b"
server_key: "95234c158844849e6da59fb0c8f831a9"
server_key_iv: "a46c996e65c4d2ab90ad9ec7c49dd56d"
server_key_version: "encryptionkey@0"
I20260927 16:43:08.469823 22344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:43:08.482733 22344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:43:08.485265 22344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:43:08.485451 22344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:43:08.492045 22344 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40667
I20260927 16:43:08.492074 22411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40667 every 8 connection(s)
I20260927 16:43:08.492902 22344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:43:08.495409 22412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:43:08.500263 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 22344
I20260927 16:43:08.500360 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:43:08.500488 22412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 678ce655693248d39b959642c5d8b567: Bootstrap starting.
I20260927 16:43:08.500665 392 external_mini_cluster.cc:1428] Setting key bf09663fa26eaeb4478fb59ae2d21b83
I20260927 16:43:08.503033 22412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 678ce655693248d39b959642c5d8b567: Neither blocks nor log segments found. Creating new log.
I20260927 16:43:08.503872 22412 log.cc:824] T 00000000000000000000000000000000 P 678ce655693248d39b959642c5d8b567: Log is configured to *not* fsync() on all Append() calls
I20260927 16:43:08.505906 22412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 678ce655693248d39b959642c5d8b567: No bootstrap required, opened a new log
Sep 27 16:43:08 dist-test-slave-z40b krb5kdc[22331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:43:08.509975 22412 raft_consensus.cc:348] T 00000000000000000000000000000000 P 678ce655693248d39b959642c5d8b567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678ce655693248d39b959642c5d8b567" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40667 } }
I20260927 16:43:08.510231 22412 raft_consensus.cc:374] T 00000000000000000000000000000000 P 678ce655693248d39b959642c5d8b567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:43:08.510308 22412 raft_consensus.cc:729] T 00000000000000000000000000000000 P 678ce655693248d39b959642c5d8b567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 678ce655693248d39b959642c5d8b567, State: Initialized, Role: FOLLOWER
I20260927 16:43:08.510773 22412 consensus_queue.cc:260] T 00000000000000000000000000000000 P 678ce655693248d39b959642c5d8b567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678ce655693248d39b959642c5d8b567" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40667 } }
I20260927 16:43:08.510934 22412 raft_consensus.cc:388] T 00000000000000000000000000000000 P 678ce655693248d39b959642c5d8b567 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:43:08.511016 22412 raft_consensus.cc:482] T 00000000000000000000000000000000 P 678ce655693248d39b959642c5d8b567 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:43:08.511148 22412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 678ce655693248d39b959642c5d8b567 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:43:08.512037 22412 raft_consensus.cc:504] T 00000000000000000000000000000000 P 678ce655693248d39b959642c5d8b567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678ce655693248d39b959642c5d8b567" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40667 } }
I20260927 16:43:08.512356 22412 leader_election.cc:302] T 00000000000000000000000000000000 P 678ce655693248d39b959642c5d8b567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 678ce655693248d39b959642c5d8b567; no voters:
I20260927 16:43:08.512667 22412 leader_election.cc:288] T 00000000000000000000000000000000 P 678ce655693248d39b959642c5d8b567 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:43:08.512722 22417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 678ce655693248d39b959642c5d8b567 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:43:08.512948 22417 raft_consensus.cc:686] T 00000000000000000000000000000000 P 678ce655693248d39b959642c5d8b567 [term 1 LEADER]: Becoming Leader. State: Replica: 678ce655693248d39b959642c5d8b567, State: Running, Role: LEADER
I20260927 16:43:08.513242 22417 consensus_queue.cc:237] T 00000000000000000000000000000000 P 678ce655693248d39b959642c5d8b567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678ce655693248d39b959642c5d8b567" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40667 } }
I20260927 16:43:08.513818 22412 sys_catalog.cc:564] T 00000000000000000000000000000000 P 678ce655693248d39b959642c5d8b567 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:43:08.514710 22418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 678ce655693248d39b959642c5d8b567 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "678ce655693248d39b959642c5d8b567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678ce655693248d39b959642c5d8b567" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40667 } } }
I20260927 16:43:08.514969 22419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 678ce655693248d39b959642c5d8b567 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 678ce655693248d39b959642c5d8b567. Latest consensus state: current_term: 1 leader_uuid: "678ce655693248d39b959642c5d8b567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678ce655693248d39b959642c5d8b567" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40667 } } }
I20260927 16:43:08.515081 22418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 678ce655693248d39b959642c5d8b567 [sys.catalog]: This master's current role is: LEADER
I20260927 16:43:08.515184 22419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 678ce655693248d39b959642c5d8b567 [sys.catalog]: This master's current role is: LEADER
I20260927 16:43:08.515516 22427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:43:08.516175 22415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:08.501929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46821 (local address 127.0.98.62:40667)
0927 16:43:08.502350 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:08.502369 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:08.502408 (+ 39us) server_negotiation.cc:407] Connection header received
0927 16:43:08.503289 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:08.503316 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:08.503721 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:08.504069 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:08.504928 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:08.506264 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:08.507138 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:08.507857 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:08.508011 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:08.510714 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:08.510719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:08.510730 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:08.510754 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:08.512598 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:08.513085 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:08.513090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:08.513094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:08.513215 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:08.513504 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:08.513507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:08.513508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:08.513855 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:08.514071 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:08.514781 (+ 710us) server_negotiation.cc:299] Negotiation successful
0927 16:43:08.515071 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":121,"threads_started":1}
I20260927 16:43:08.518502 22427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:43:08.519011 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:43:08.524482 22427 catalog_manager.cc:1347] Generated new cluster ID: e7a9842328a048549faddf5f48960825
I20260927 16:43:08.524539 22427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:43:08.529223 22415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:08.521106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58031 (local address 127.0.98.62:40667)
0927 16:43:08.521216 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:08.521222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:08.521442 (+ 220us) server_negotiation.cc:407] Connection header received
0927 16:43:08.521533 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:08.521536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:08.521583 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:08.521664 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:08.522391 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:08.523196 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:08.524371 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:08.524692 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:08.524808 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:08.526102 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:08.526105 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:08.526108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:08.526130 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:08.527655 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:08.528071 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:08.528075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:08.528077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:08.528157 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:08.528419 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:08.528424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:08.528426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:08.528595 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:08.528724 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:08.528953 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 16:43:08.529059 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 16:43:08.537839 22427 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:43:08.539147 22427 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:43:08.544471 22427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 678ce655693248d39b959642c5d8b567: Generated new TSK 0
I20260927 16:43:08.545264 22427 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:43:08.549736 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 22344
2026-09-27T16:43:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1349 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:43:08 dist-test-slave-z40b krb5kdc[22443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:43:08 dist-test-slave-z40b krb5kdc[22443](info): set up 2 sockets
Sep 27 16:43:08 dist-test-slave-z40b krb5kdc[22443](info): commencing operation
krb5kdc: starting...
W20260927 16:43:09.567484 392 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:43:09 dist-test-slave-z40b krb5kdc[22443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:43:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:43:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:43:09.679766 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35109
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:43:09.783829 22456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:43:09.784040 22456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:43:09.784077 22456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:43:09.787767 22456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:43:09.787813 22456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:43:09.787832 22456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:43:09.787849 22456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:43:09.787863 22456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:43:09.792665 22456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:43:09.793459 22456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:43:09.794296 22456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:43:09.802162 22464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:43:09.802304 22456 server_base.cc:1034] running on GCE node
W20260927 16:43:09.802212 22461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:43:09.802568 22462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:43:09.803112 22456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:43:09.804255 22456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:43:09.805408 22456 hybrid_clock.cc:648] HybridClock initialized: now 1790527389805391 us; error 29 us; skew 500 ppm
Sep 27 16:43:09 dist-test-slave-z40b krb5kdc[22443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:43:09.808121 22456 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:43:09.808866 22456 webserver.cc:460] Webserver started at http://127.0.98.62:45691/ using document root <none> and password file <none>
I20260927 16:43:09.809335 22456 fs_manager.cc:359] Metadata directory not provided
I20260927 16:43:09.809388 22456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:43:09.809535 22456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:43:09.811470 22456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "a1ebe24235ad45fbb2fe0c406a572681"
format_stamp: "Formatted at 2026-09-27 16:43:09 on dist-test-slave-z40b"
server_key: "cffb5c176ac013815bc0434243682644"
server_key_iv: "6218216ec96eac605a053204938db3be"
server_key_version: "encryptionkey@0"
I20260927 16:43:09.811955 22456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "a1ebe24235ad45fbb2fe0c406a572681"
format_stamp: "Formatted at 2026-09-27 16:43:09 on dist-test-slave-z40b"
server_key: "cffb5c176ac013815bc0434243682644"
server_key_iv: "6218216ec96eac605a053204938db3be"
server_key_version: "encryptionkey@0"
I20260927 16:43:09.815346 22456 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:43:09.817513 22471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:43:09.818655 22456 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:43:09.818819 22456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "a1ebe24235ad45fbb2fe0c406a572681"
format_stamp: "Formatted at 2026-09-27 16:43:09 on dist-test-slave-z40b"
server_key: "cffb5c176ac013815bc0434243682644"
server_key_iv: "6218216ec96eac605a053204938db3be"
server_key_version: "encryptionkey@0"
I20260927 16:43:09.818925 22456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:43:09.830216 22456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:43:09.833319 22456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:43:09.833501 22456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:43:09.840031 22456 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35109
I20260927 16:43:09.840056 22523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35109 every 8 connection(s)
I20260927 16:43:09.840884 22456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:43:09.843530 22524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:43:09.845811 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 22456
I20260927 16:43:09.845942 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:43:09.846253 392 external_mini_cluster.cc:1428] Setting key e5d1763d40ea39ab71ea696869420c6e
I20260927 16:43:09.849819 22524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1ebe24235ad45fbb2fe0c406a572681: Bootstrap starting.
I20260927 16:43:09.852335 22524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1ebe24235ad45fbb2fe0c406a572681: Neither blocks nor log segments found. Creating new log.
I20260927 16:43:09.853209 22524 log.cc:824] T 00000000000000000000000000000000 P a1ebe24235ad45fbb2fe0c406a572681: Log is configured to *not* fsync() on all Append() calls
I20260927 16:43:09.855373 22524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1ebe24235ad45fbb2fe0c406a572681: No bootstrap required, opened a new log
I20260927 16:43:09.856158 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:09.847765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43665 (local address 127.0.98.62:35109)
0927 16:43:09.848164 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:09.848182 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:09.848224 (+ 42us) server_negotiation.cc:407] Connection header received
0927 16:43:09.849051 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:09.849073 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:09.849484 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:09.849829 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:09.850211 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:09.851705 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:09.852671 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:09.853346 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:09.853501 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:09.854261 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:09.854266 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:09.854281 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:43:09.854753 (+ 472us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:09.854829 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:43:09.854859 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:09.854956 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:09.855055 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 16:43:09.855276 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":103,"threads_started":1}
I20260927 16:43:09.858820 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:43:09.859475 22524 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1ebe24235ad45fbb2fe0c406a572681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1ebe24235ad45fbb2fe0c406a572681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35109 } }
I20260927 16:43:09.859675 22524 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1ebe24235ad45fbb2fe0c406a572681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:43:09.859731 22524 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1ebe24235ad45fbb2fe0c406a572681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1ebe24235ad45fbb2fe0c406a572681, State: Initialized, Role: FOLLOWER
I20260927 16:43:09.860222 22524 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1ebe24235ad45fbb2fe0c406a572681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1ebe24235ad45fbb2fe0c406a572681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35109 } }
I20260927 16:43:09.860352 22524 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1ebe24235ad45fbb2fe0c406a572681 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:43:09.860436 22524 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1ebe24235ad45fbb2fe0c406a572681 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:43:09.860545 22524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1ebe24235ad45fbb2fe0c406a572681 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:43:09.861559 22524 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1ebe24235ad45fbb2fe0c406a572681 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1ebe24235ad45fbb2fe0c406a572681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35109 } }
I20260927 16:43:09.861883 22524 leader_election.cc:302] T 00000000000000000000000000000000 P a1ebe24235ad45fbb2fe0c406a572681 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1ebe24235ad45fbb2fe0c406a572681; no voters:
I20260927 16:43:09.862155 22524 leader_election.cc:288] T 00000000000000000000000000000000 P a1ebe24235ad45fbb2fe0c406a572681 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:43:09.862251 22534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1ebe24235ad45fbb2fe0c406a572681 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:43:09.862465 22534 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1ebe24235ad45fbb2fe0c406a572681 [term 1 LEADER]: Becoming Leader. State: Replica: a1ebe24235ad45fbb2fe0c406a572681, State: Running, Role: LEADER
I20260927 16:43:09.862839 22534 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1ebe24235ad45fbb2fe0c406a572681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1ebe24235ad45fbb2fe0c406a572681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35109 } }
I20260927 16:43:09.863365 22524 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1ebe24235ad45fbb2fe0c406a572681 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:43:09.864470 22535 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1ebe24235ad45fbb2fe0c406a572681 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1ebe24235ad45fbb2fe0c406a572681" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1ebe24235ad45fbb2fe0c406a572681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35109 } } }
I20260927 16:43:09.864614 22535 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1ebe24235ad45fbb2fe0c406a572681 [sys.catalog]: This master's current role is: LEADER
I20260927 16:43:09.864890 22535 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1ebe24235ad45fbb2fe0c406a572681 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1ebe24235ad45fbb2fe0c406a572681. Latest consensus state: current_term: 1 leader_uuid: "a1ebe24235ad45fbb2fe0c406a572681" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1ebe24235ad45fbb2fe0c406a572681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35109 } } }
I20260927 16:43:09.865044 22535 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1ebe24235ad45fbb2fe0c406a572681 [sys.catalog]: This master's current role is: LEADER
I20260927 16:43:09.865367 22543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:43:09.865607 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:09.860959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49177 (local address 127.0.98.62:35109)
0927 16:43:09.861069 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:09.861075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:09.861249 (+ 174us) server_negotiation.cc:407] Connection header received
0927 16:43:09.861378 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:09.861381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:09.861425 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:09.861506 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:09.861745 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:09.862420 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:09.863532 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:09.863893 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:09.863991 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:09.865011 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:09.865015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:09.865017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:43:09.865030 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:09.865060 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:43:09.865074 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:09.865139 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:09.865450 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 16:43:09.865497 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 16:43:09.868800 22543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:43:09.873925 22543 catalog_manager.cc:1347] Generated new cluster ID: 3004de856f1248e99da8e57af226fab6
I20260927 16:43:09.873992 22543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:43:09.884418 22543 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:43:09.885249 22543 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:43:09.893222 22543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1ebe24235ad45fbb2fe0c406a572681: Generated new TSK 0
I20260927 16:43:09.893806 22543 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:43:09.903349 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 22456
2026-09-27T16:43:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:43:09 dist-test-slave-z40b krb5kdc[22555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:43:09 dist-test-slave-z40b krb5kdc[22555](info): set up 2 sockets
Sep 27 16:43:09 dist-test-slave-z40b krb5kdc[22555](info): commencing operation
krb5kdc: starting...
W20260927 16:43:10.919416 392 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:43:10 dist-test-slave-z40b krb5kdc[22555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:43:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:43:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:43:11.034408 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44115
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:43:11.137820 22568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:43:11.138085 22568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:43:11.138177 22568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:43:11.142048 22568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:43:11.142158 22568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:43:11.142196 22568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:43:11.142231 22568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:43:11.142261 22568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:43:11.147989 22568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:43:11.149022 22568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:43:11.150027 22568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:43:11.158192 22573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:43:11.158183 22574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:43:11.158257 22576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:43:11.158417 22568 server_base.cc:1034] running on GCE node
I20260927 16:43:11.159003 22568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:43:11.160092 22568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:43:11.161281 22568 hybrid_clock.cc:648] HybridClock initialized: now 1790527391161255 us; error 46 us; skew 500 ppm
Sep 27 16:43:11 dist-test-slave-z40b krb5kdc[22555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:43:11.163895 22568 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:43:11.164723 22568 webserver.cc:460] Webserver started at http://127.0.98.62:34051/ using document root <none> and password file <none>
I20260927 16:43:11.165244 22568 fs_manager.cc:359] Metadata directory not provided
I20260927 16:43:11.165338 22568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:43:11.165556 22568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:43:11.167549 22568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "b72f82ff5eff4193b1b03078dd870016"
format_stamp: "Formatted at 2026-09-27 16:43:11 on dist-test-slave-z40b"
server_key: "432283c384ec1060816502143a0c7169"
server_key_iv: "50c5760cf8c4d7b7982d66bd343d392a"
server_key_version: "encryptionkey@0"
I20260927 16:43:11.168093 22568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "b72f82ff5eff4193b1b03078dd870016"
format_stamp: "Formatted at 2026-09-27 16:43:11 on dist-test-slave-z40b"
server_key: "432283c384ec1060816502143a0c7169"
server_key_iv: "50c5760cf8c4d7b7982d66bd343d392a"
server_key_version: "encryptionkey@0"
I20260927 16:43:11.171540 22568 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:43:11.173817 22583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:43:11.174925 22568 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:43:11.175061 22568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "b72f82ff5eff4193b1b03078dd870016"
format_stamp: "Formatted at 2026-09-27 16:43:11 on dist-test-slave-z40b"
server_key: "432283c384ec1060816502143a0c7169"
server_key_iv: "50c5760cf8c4d7b7982d66bd343d392a"
server_key_version: "encryptionkey@0"
I20260927 16:43:11.175180 22568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:43:11.179457 22568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:43:11.182425 22568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:43:11.182615 22568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:43:11.189338 22568 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44115
I20260927 16:43:11.189363 22635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44115 every 8 connection(s)
I20260927 16:43:11.190136 22568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:43:11.192636 22636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:43:11.197749 22636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b72f82ff5eff4193b1b03078dd870016: Bootstrap starting.
I20260927 16:43:11.199994 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 22568
I20260927 16:43:11.200141 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:43:11.200313 22636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b72f82ff5eff4193b1b03078dd870016: Neither blocks nor log segments found. Creating new log.
I20260927 16:43:11.200448 392 external_mini_cluster.cc:1428] Setting key 6908a9e9aec63a4aab4f283e10265b43
I20260927 16:43:11.201097 22636 log.cc:824] T 00000000000000000000000000000000 P b72f82ff5eff4193b1b03078dd870016: Log is configured to *not* fsync() on all Append() calls
I20260927 16:43:11.202966 22636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b72f82ff5eff4193b1b03078dd870016: No bootstrap required, opened a new log
I20260927 16:43:11.205060 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:11.201524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56341 (local address 127.0.98.62:44115)
0927 16:43:11.201905 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:11.201920 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:11.201953 (+ 33us) server_negotiation.cc:407] Connection header received
0927 16:43:11.202558 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:11.202573 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:11.202827 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:11.203159 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:11.203938 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:11.203942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:11.203970 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:43:11.203979 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:11.204024 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:43:11.204041 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:11.204126 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:11.204255 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 16:43:11.204415 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":96,"threads_started":1}
I20260927 16:43:11.206916 22636 raft_consensus.cc:348] T 00000000000000000000000000000000 P b72f82ff5eff4193b1b03078dd870016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72f82ff5eff4193b1b03078dd870016" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44115 } }
I20260927 16:43:11.207158 22636 raft_consensus.cc:374] T 00000000000000000000000000000000 P b72f82ff5eff4193b1b03078dd870016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:43:11.207247 22636 raft_consensus.cc:729] T 00000000000000000000000000000000 P b72f82ff5eff4193b1b03078dd870016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b72f82ff5eff4193b1b03078dd870016, State: Initialized, Role: FOLLOWER
I20260927 16:43:11.207717 22636 consensus_queue.cc:260] T 00000000000000000000000000000000 P b72f82ff5eff4193b1b03078dd870016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72f82ff5eff4193b1b03078dd870016" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44115 } }
I20260927 16:43:11.207885 22636 raft_consensus.cc:388] T 00000000000000000000000000000000 P b72f82ff5eff4193b1b03078dd870016 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:43:11.207902 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:43:11.207962 22636 raft_consensus.cc:482] T 00000000000000000000000000000000 P b72f82ff5eff4193b1b03078dd870016 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:43:11.208050 22636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b72f82ff5eff4193b1b03078dd870016 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:43:11.208882 22636 raft_consensus.cc:504] T 00000000000000000000000000000000 P b72f82ff5eff4193b1b03078dd870016 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72f82ff5eff4193b1b03078dd870016" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44115 } }
I20260927 16:43:11.209229 22636 leader_election.cc:302] T 00000000000000000000000000000000 P b72f82ff5eff4193b1b03078dd870016 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b72f82ff5eff4193b1b03078dd870016; no voters:
I20260927 16:43:11.209553 22636 leader_election.cc:288] T 00000000000000000000000000000000 P b72f82ff5eff4193b1b03078dd870016 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:43:11.209640 22645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b72f82ff5eff4193b1b03078dd870016 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:43:11.209928 22645 raft_consensus.cc:686] T 00000000000000000000000000000000 P b72f82ff5eff4193b1b03078dd870016 [term 1 LEADER]: Becoming Leader. State: Replica: b72f82ff5eff4193b1b03078dd870016, State: Running, Role: LEADER
I20260927 16:43:11.210261 22645 consensus_queue.cc:237] T 00000000000000000000000000000000 P b72f82ff5eff4193b1b03078dd870016 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72f82ff5eff4193b1b03078dd870016" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44115 } }
I20260927 16:43:11.210820 22636 sys_catalog.cc:564] T 00000000000000000000000000000000 P b72f82ff5eff4193b1b03078dd870016 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:43:11.211886 22646 sys_catalog.cc:455] T 00000000000000000000000000000000 P b72f82ff5eff4193b1b03078dd870016 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b72f82ff5eff4193b1b03078dd870016" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72f82ff5eff4193b1b03078dd870016" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44115 } } }
I20260927 16:43:11.212011 22646 sys_catalog.cc:458] T 00000000000000000000000000000000 P b72f82ff5eff4193b1b03078dd870016 [sys.catalog]: This master's current role is: LEADER
I20260927 16:43:11.213183 22647 sys_catalog.cc:455] T 00000000000000000000000000000000 P b72f82ff5eff4193b1b03078dd870016 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b72f82ff5eff4193b1b03078dd870016. Latest consensus state: current_term: 1 leader_uuid: "b72f82ff5eff4193b1b03078dd870016" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72f82ff5eff4193b1b03078dd870016" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44115 } } }
I20260927 16:43:11.213428 22647 sys_catalog.cc:458] T 00000000000000000000000000000000 P b72f82ff5eff4193b1b03078dd870016 [sys.catalog]: This master's current role is: LEADER
I20260927 16:43:11.213554 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:11.210310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59535 (local address 127.0.98.62:44115)
0927 16:43:11.210426 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:11.210430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:11.211546 (+ 1116us) server_negotiation.cc:407] Connection header received
0927 16:43:11.211774 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:11.211777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:11.211817 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:11.211872 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:11.212715 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:11.212718 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:11.212721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:43:11.212732 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:11.212756 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:43:11.212768 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:11.212838 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:11.213343 (+ 505us) server_negotiation.cc:299] Negotiation successful
0927 16:43:11.213383 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 16:43:11.215592 22661 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b72f82ff5eff4193b1b03078dd870016: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:43:11.215678 22661 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:43:11.215746 22655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:43:11.216387 22655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:43:11.222093 22655 catalog_manager.cc:1347] Generated new cluster ID: 2dda70837ef2416e9de098209c76b1b9
I20260927 16:43:11.222150 22655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:43:11.247601 22655 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:43:11.247892 22655 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:43:11.252199 22655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b72f82ff5eff4193b1b03078dd870016: Generated new TSK 0
I20260927 16:43:11.252813 22655 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:43:11.259140 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 22568
2026-09-27T16:43:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1355 ms)
[----------] 18 tests from AuthTokenIssuingTest (24258 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:43:11 dist-test-slave-z40b krb5kdc[22667](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:43:11 dist-test-slave-z40b krb5kdc[22667](info): set up 2 sockets
Sep 27 16:43:11 dist-test-slave-z40b krb5kdc[22667](info): commencing operation
krb5kdc: starting...
W20260927 16:43:12.267436 392 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:43:12 dist-test-slave-z40b krb5kdc[22667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:43:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:43:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:43:12.381699 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39069
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39069,127.0.98.61:33043,127.0.98.60:43539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:43:12.485534 22680 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 16:43:12.485782 22680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:43:12.485805 22680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:43:12.485841 22680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:43:12.489704 22680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:43:12.489746 22680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:43:12.489764 22680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:43:12.489779 22680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:43:12.489794 22680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:43:12.494767 22680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39069,127.0.98.61:33043,127.0.98.60:43539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:43:12.495726 22680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:43:12.496646 22680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:43:12.504889 22685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:43:12.504889 22686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:43:12.505025 22680 server_base.cc:1034] running on GCE node
W20260927 16:43:12.505249 22688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:43:12.505810 22680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:43:12.506946 22680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:43:12.508136 22680 hybrid_clock.cc:648] HybridClock initialized: now 1790527392508128 us; error 45 us; skew 500 ppm
Sep 27 16:43:12 dist-test-slave-z40b krb5kdc[22667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:43:12.510680 22680 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:43:12.511570 22680 webserver.cc:460] Webserver started at http://127.0.98.62:41819/ using document root <none> and password file <none>
I20260927 16:43:12.512089 22680 fs_manager.cc:359] Metadata directory not provided
I20260927 16:43:12.512179 22680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:43:12.512406 22680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:43:12.514325 22680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "0dea43d0be22409fa79bb91bc3c2e50b"
format_stamp: "Formatted at 2026-09-27 16:43:12 on dist-test-slave-z40b"
server_key: "f530ddaf3cd9cf1ca4a9a279a61df146"
server_key_iv: "b4f5cf573f1fe75c6cf0ec42acc4c7d4"
server_key_version: "encryptionkey@0"
I20260927 16:43:12.514858 22680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "0dea43d0be22409fa79bb91bc3c2e50b"
format_stamp: "Formatted at 2026-09-27 16:43:12 on dist-test-slave-z40b"
server_key: "f530ddaf3cd9cf1ca4a9a279a61df146"
server_key_iv: "b4f5cf573f1fe75c6cf0ec42acc4c7d4"
server_key_version: "encryptionkey@0"
I20260927 16:43:12.518430 22680 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:43:12.520705 22695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:43:12.521731 22680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:43:12.521855 22680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "0dea43d0be22409fa79bb91bc3c2e50b"
format_stamp: "Formatted at 2026-09-27 16:43:12 on dist-test-slave-z40b"
server_key: "f530ddaf3cd9cf1ca4a9a279a61df146"
server_key_iv: "b4f5cf573f1fe75c6cf0ec42acc4c7d4"
server_key_version: "encryptionkey@0"
I20260927 16:43:12.521950 22680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:43:12.534912 22680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:43:12.537441 22680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:43:12.537637 22680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:43:12.544190 22680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39069
I20260927 16:43:12.544453 22747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39069 every 8 connection(s)
I20260927 16:43:12.545091 22680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:43:12.547348 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 22680
I20260927 16:43:12.547492 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:43:12.547772 392 external_mini_cluster.cc:1428] Setting key df1af78516f3e5368e8388538c37db6c
I20260927 16:43:12.547819 22748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:43:12.551381 22748 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39069 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 16:43:12 dist-test-slave-z40b krb5kdc[22667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:43:12.565820 22751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.553455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40030 (local address 127.0.98.62:39069)
0927 16:43:12.553842 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:12.553860 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:12.554120 (+ 260us) server_negotiation.cc:407] Connection header received
0927 16:43:12.555056 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:12.555060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:12.555283 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:12.555461 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:12.556209 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.557175 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.558157 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.558515 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.558679 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:12.561418 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:12.561423 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:12.561431 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:12.561462 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:12.562962 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:12.563461 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:12.563473 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:12.563475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:12.563567 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:12.563858 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:12.563864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:12.563865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:12.564183 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:12.564326 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:12.564655 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 16:43:12.564879 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":125,"threads_started":1}
I20260927 16:43:12.565820 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.553455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39069 (local address 127.0.0.1:40030)
0927 16:43:12.553981 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:12.554014 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 16:43:12.554774 (+ 760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:43:12.555458 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:43:12.555462 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:43:12.555851 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:43:12.556073 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:12.556085 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.557302 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.557305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:12.558046 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:12.558055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.558664 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.558667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:12.558776 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:12.559436 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:43:12.559461 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:43:12.561246 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:43:12.563066 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:12.563075 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:12.563084 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:12.563361 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:12.563659 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:12.563663 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:12.563665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:12.563771 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:12.564307 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:43:12.564313 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:43:12.564443 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 16:43:12.564601 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 16:43:12.564950 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":103,"spinlock_wait_cycles":25472,"thread_start_us":132,"threads_started":1}
I20260927 16:43:12.568048 22748 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33043 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:43:12.568822 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.568519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33043 (local address 127.0.0.1:39288)
0927 16:43:12.568649 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:12.568749 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 16:43:12.569034 22697 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33043: connect: Connection refused (error 111)
I20260927 16:43:12.569774 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.569566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33043 (local address 127.0.0.1:39298)
0927 16:43:12.569683 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:12.569724 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 16:43:12.569947 22748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33043: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33043: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 16:43:12.573766 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33043
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39069,127.0.98.61:33043,127.0.98.60:43539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:43:12.619253 22748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33043 } attempt: 1
I20260927 16:43:12.620183 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.619896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33043 (local address 127.0.0.1:39312)
0927 16:43:12.620051 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:12.620111 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 16:43:12.620990 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.620779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33043 (local address 127.0.0.1:39316)
0927 16:43:12.620886 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:12.620928 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 16:43:12.621157 22748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33043: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33043: connect: Connection refused (error 111)
W20260927 16:43:12.680110 22755 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 16:43:12.680357 22755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:43:12.680378 22755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:43:12.680413 22755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:43:12.684185 22755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:43:12.684232 22755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:43:12.684252 22755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:43:12.684267 22755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:43:12.684283 22755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:43:12.689276 22755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39069,127.0.98.61:33043,127.0.98.60:43539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:43:12.689311 22748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33043 } attempt: 2
I20260927 16:43:12.690127 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.689835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33043 (local address 127.0.0.1:39320)
0927 16:43:12.689972 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:12.690041 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 16:43:12.690264 22755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:43:12.690889 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.690699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33043 (local address 127.0.0.1:39322)
0927 16:43:12.690800 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:12.690842 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 16:43:12.691078 22748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33043: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33043: connect: Connection refused (error 111)
I20260927 16:43:12.691443 22755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:43:12.699973 22761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:43:12.699965 22763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:43:12.699990 22760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:43:12.700415 22755 server_base.cc:1034] running on GCE node
I20260927 16:43:12.700958 22755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:43:12.701957 22755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:43:12.703153 22755 hybrid_clock.cc:648] HybridClock initialized: now 1790527392703139 us; error 30 us; skew 500 ppm
Sep 27 16:43:12 dist-test-slave-z40b krb5kdc[22667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:43:12.705770 22755 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 16:43:12.706566 22755 webserver.cc:460] Webserver started at http://127.0.98.61:33895/ using document root <none> and password file <none>
I20260927 16:43:12.707108 22755 fs_manager.cc:359] Metadata directory not provided
I20260927 16:43:12.707195 22755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:43:12.707443 22755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:43:12.709359 22755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-1/data/instance:
uuid: "65a36ddcc92d4a4db8a33940a1d9ffb0"
format_stamp: "Formatted at 2026-09-27 16:43:12 on dist-test-slave-z40b"
server_key: "b5fa0be69f69c4bffe17f1e9b67a7157"
server_key_iv: "7569ade5ef939887b27369c98344dfef"
server_key_version: "encryptionkey@0"
I20260927 16:43:12.709939 22755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-1/wal/instance:
uuid: "65a36ddcc92d4a4db8a33940a1d9ffb0"
format_stamp: "Formatted at 2026-09-27 16:43:12 on dist-test-slave-z40b"
server_key: "b5fa0be69f69c4bffe17f1e9b67a7157"
server_key_iv: "7569ade5ef939887b27369c98344dfef"
server_key_version: "encryptionkey@0"
I20260927 16:43:12.713428 22755 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260927 16:43:12.715720 22770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:43:12.716849 22755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:43:12.716977 22755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-1/wal
uuid: "65a36ddcc92d4a4db8a33940a1d9ffb0"
format_stamp: "Formatted at 2026-09-27 16:43:12 on dist-test-slave-z40b"
server_key: "b5fa0be69f69c4bffe17f1e9b67a7157"
server_key_iv: "7569ade5ef939887b27369c98344dfef"
server_key_version: "encryptionkey@0"
I20260927 16:43:12.717079 22755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:43:12.733347 22755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:43:12.736390 22755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:43:12.736586 22755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:43:12.743379 22755 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33043
I20260927 16:43:12.743649 22822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33043 every 8 connection(s)
I20260927 16:43:12.744324 22755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-1/data/info.pb
I20260927 16:43:12.746939 22823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:43:12.749233 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 22755
I20260927 16:43:12.749365 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-1/wal/instance
I20260927 16:43:12.749656 392 external_mini_cluster.cc:1428] Setting key 9fd021ccb543ee95d43ddbc39c505b7d
I20260927 16:43:12.751493 22823 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39069 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 16:43:12 dist-test-slave-z40b krb5kdc[22667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:43:12.765995 22751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.753310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40040 (local address 127.0.98.62:39069)
0927 16:43:12.753493 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:12.753500 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:12.754286 (+ 786us) server_negotiation.cc:407] Connection header received
0927 16:43:12.755648 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:12.755650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:12.755696 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:12.755770 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:12.757269 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.757768 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.759197 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.759650 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.759757 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:12.762133 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:12.762136 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:12.762138 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:12.762161 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:12.763828 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:12.764294 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:12.764300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:12.764304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:12.764394 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:12.764767 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:12.764772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:12.764776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:12.765028 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:12.765135 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:12.765606 (+ 471us) server_negotiation.cc:299] Negotiation successful
0927 16:43:12.765758 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 16:43:12.766520 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.753656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39069 (local address 127.0.0.1:40040)
0927 16:43:12.754128 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:12.754177 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 16:43:12.755400 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:43:12.755969 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:43:12.755978 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:43:12.756451 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:43:12.757116 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:12.757137 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.757898 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.757904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:12.759077 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:12.759087 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.759798 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.759801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:12.759936 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:12.760547 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:43:12.760564 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:43:12.761968 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:43:12.763942 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:12.763947 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:12.763957 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:12.764192 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:12.764488 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:12.764490 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:12.764492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:12.764613 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:12.765134 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:43:12.765138 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:43:12.765353 (+ 215us) client_negotiation.cc:769] Sending connection context
0927 16:43:12.765501 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 16:43:12.765722 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":144,"threads_started":1}
I20260927 16:43:12.768054 22823 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33043 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:43:12 dist-test-slave-z40b krb5kdc[22667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 16:43:12.776636 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.768548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33043 (local address 127.0.0.1:39338)
0927 16:43:12.768636 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:12.768650 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:43:12.768718 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:43:12.769276 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:43:12.769279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:43:12.769452 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:43:12.769540 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:12.769545 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.770571 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.770573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:12.771052 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:12.771059 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.771655 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.771657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:12.771725 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:12.772201 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:43:12.772212 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:43:12.773454 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:43:12.775247 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:12.775252 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:12.775255 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:12.775516 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:12.775791 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:12.775794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:12.775796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:12.775867 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:12.776303 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:43:12.776305 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:43:12.776372 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 16:43:12.776451 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 16:43:12.776533 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 16:43:12.776935 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.768689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39338 (local address 127.0.98.61:33043)
0927 16:43:12.768966 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:12.768975 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:12.768992 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:43:12.769026 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:12.769029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:12.769163 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:12.769267 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:12.769636 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.770469 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.771176 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.771561 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.771642 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:12.773598 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:12.773606 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:12.773615 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:12.773634 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:12.775070 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:12.775618 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:12.775625 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:12.775627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:12.775710 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:12.775941 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:12.775944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:12.775944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:12.776209 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:12.776308 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:12.776581 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 16:43:12.776685 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":93,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 16:43:12.777920 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43539
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39069,127.0.98.61:33043,127.0.98.60:43539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:43:12.778791 22823 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:43:12.779495 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.779277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43539 (local address 127.0.0.1:57006)
0927 16:43:12.779373 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:12.779435 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 16:43:12.779731 22773 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43539: connect: Connection refused (error 111)
I20260927 16:43:12.780550 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.780360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43539 (local address 127.0.0.1:57012)
0927 16:43:12.780458 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:12.780507 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 16:43:12.780742 22823 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43539: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43539: connect: Connection refused (error 111)
I20260927 16:43:12.782250 22748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33043 } attempt: 3
Sep 27 16:43:12 dist-test-slave-z40b krb5kdc[22667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 16:43:12.791253 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.782618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33043 (local address 127.0.0.1:39348)
0927 16:43:12.782772 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:12.782789 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:43:12.782878 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:43:12.783113 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:43:12.783119 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:43:12.783348 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:43:12.783515 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:12.783525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.784246 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.784252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:12.785807 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:12.785822 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.786275 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.786278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:12.786420 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:12.786859 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:43:12.786880 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:43:12.788206 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:43:12.789973 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:12.789975 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:12.789977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:12.790152 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:12.790439 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:12.790441 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:12.790442 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:12.790524 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:12.790870 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:43:12.790872 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:43:12.790940 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 16:43:12.791012 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 16:43:12.791098 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 16:43:12.791348 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.782707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39348 (local address 127.0.98.61:33043)
0927 16:43:12.782809 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:12.782812 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:12.782828 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:43:12.782969 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:12.782971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:12.783000 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:12.783059 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:12.783629 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.784085 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.785939 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.786161 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.786234 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:12.788343 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:12.788346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:12.788348 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:12.788367 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:12.789865 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:12.790242 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:12.790247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:12.790249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:12.790355 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:12.790602 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:12.790605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:12.790606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:12.790787 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:12.790878 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:12.791080 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 16:43:12.791180 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 16:43:12.792526 22748 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:43:12.793129 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.792920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43539 (local address 127.0.0.1:57018)
0927 16:43:12.793020 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:12.793066 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 16:43:12.793843 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.793616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43539 (local address 127.0.0.1:57032)
0927 16:43:12.793764 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:12.793802 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260927 16:43:12.794019 22748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43539: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43539: connect: Connection refused (error 111)
I20260927 16:43:12.825181 22748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 } attempt: 1
I20260927 16:43:12.826076 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.825770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43539 (local address 127.0.0.1:57042)
0927 16:43:12.825900 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:12.825978 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 16:43:12.827026 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.826814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43539 (local address 127.0.0.1:57058)
0927 16:43:12.826919 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:12.826966 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 16:43:12.827224 22748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43539: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43539: connect: Connection refused (error 111)
I20260927 16:43:12.829972 22823 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 } attempt: 1
I20260927 16:43:12.830727 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.830507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43539 (local address 127.0.0.1:57062)
0927 16:43:12.830622 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:12.830668 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 16:43:12.831467 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.831289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43539 (local address 127.0.0.1:57072)
0927 16:43:12.831381 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:12.831418 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260927 16:43:12.831637 22823 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43539: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43539: connect: Connection refused (error 111)
W20260927 16:43:12.888478 22829 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 16:43:12.888713 22829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:43:12.888736 22829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:43:12.888818 22829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:43:12.893137 22829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:43:12.893182 22829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:43:12.893199 22829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:43:12.893240 22829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:43:12.893270 22829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:43:12.898073 22829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39069,127.0.98.61:33043,127.0.98.60:43539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:43:12.898999 22829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:43:12.899785 22823 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 } attempt: 2
I20260927 16:43:12.900018 22829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:43:12.900550 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.900264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43539 (local address 127.0.0.1:57074)
0927 16:43:12.900422 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:12.900479 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 16:43:12.901229 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.901080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43539 (local address 127.0.0.1:57088)
0927 16:43:12.901145 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:12.901183 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260927 16:43:12.901584 22823 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43539: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43539: connect: Connection refused (error 111)
I20260927 16:43:12.902370 22748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 } attempt: 2
I20260927 16:43:12.902951 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.902725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43539 (local address 127.0.0.1:57104)
0927 16:43:12.902846 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:12.902896 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 16:43:12.903640 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.903435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43539 (local address 127.0.0.1:57118)
0927 16:43:12.903551 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:12.903596 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 16:43:12.903779 22748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43539: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43539: connect: Connection refused (error 111)
W20260927 16:43:12.908394 22835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:43:12.908394 22836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:43:12.908404 22838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:43:12.908738 22829 server_base.cc:1034] running on GCE node
I20260927 16:43:12.909360 22829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:43:12.910386 22829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:43:12.911576 22829 hybrid_clock.cc:648] HybridClock initialized: now 1790527392911564 us; error 32 us; skew 500 ppm
Sep 27 16:43:12 dist-test-slave-z40b krb5kdc[22667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:43:12.914203 22829 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 16:43:12.915033 22829 webserver.cc:460] Webserver started at http://127.0.98.60:44661/ using document root <none> and password file <none>
I20260927 16:43:12.915844 22829 fs_manager.cc:359] Metadata directory not provided
I20260927 16:43:12.915954 22829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:43:12.916184 22829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:43:12.918079 22829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-2/data/instance:
uuid: "71795024f5e94aa09573527353f00000"
format_stamp: "Formatted at 2026-09-27 16:43:12 on dist-test-slave-z40b"
server_key: "1436821dd078f3616aa50bf44a48fa18"
server_key_iv: "76ed1837f4e0106546171fa1605cf41f"
server_key_version: "encryptionkey@0"
I20260927 16:43:12.918627 22829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-2/wal/instance:
uuid: "71795024f5e94aa09573527353f00000"
format_stamp: "Formatted at 2026-09-27 16:43:12 on dist-test-slave-z40b"
server_key: "1436821dd078f3616aa50bf44a48fa18"
server_key_iv: "76ed1837f4e0106546171fa1605cf41f"
server_key_version: "encryptionkey@0"
I20260927 16:43:12.922261 22829 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:43:12.924525 22845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:43:12.925719 22829 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:43:12.925858 22829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-2/wal
uuid: "71795024f5e94aa09573527353f00000"
format_stamp: "Formatted at 2026-09-27 16:43:12 on dist-test-slave-z40b"
server_key: "1436821dd078f3616aa50bf44a48fa18"
server_key_iv: "76ed1837f4e0106546171fa1605cf41f"
server_key_version: "encryptionkey@0"
I20260927 16:43:12.925956 22829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:43:12.949203 22829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:43:12.951817 22829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:43:12.951999 22829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:43:12.958164 22829 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43539
I20260927 16:43:12.958451 22897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43539 every 8 connection(s)
I20260927 16:43:12.959071 22829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-2/data/info.pb
I20260927 16:43:12.961581 22898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:43:12.964401 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 22829
I20260927 16:43:12.964555 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790527121515808-392-0/minicluster-data/master-2/wal/instance
I20260927 16:43:12.964808 392 external_mini_cluster.cc:1428] Setting key 3e1ca837fa52d94b408f21de6062d032
I20260927 16:43:12.965076 22898 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39069 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:43:12 dist-test-slave-z40b krb5kdc[22667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 16:43:12 dist-test-slave-z40b krb5kdc[22667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 16:43:12.976913 22751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.966880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40042 (local address 127.0.98.62:39069)
0927 16:43:12.967009 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:12.967013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:12.967336 (+ 323us) server_negotiation.cc:407] Connection header received
0927 16:43:12.967719 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:12.967724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:12.967774 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:12.967873 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:12.968730 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.969204 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.970215 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.970473 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.970566 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:12.973172 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:12.973177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:12.973181 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:12.973208 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:12.974830 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:12.975380 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:12.975383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:12.975384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:12.975463 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:12.975856 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:12.975865 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:12.975867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:12.976039 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:12.976146 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:12.976616 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 16:43:12.976727 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 16:43:12.977648 22902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.966720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39069 (local address 127.0.0.1:40042)
0927 16:43:12.967163 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:12.967235 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 16:43:12.967516 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:43:12.967932 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:43:12.967937 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:43:12.968307 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:43:12.968596 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:12.968607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.969315 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.969319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:12.970103 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:12.970111 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.970617 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.970621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:12.970787 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:12.971431 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:43:12.971457 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:43:12.973033 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:43:12.974930 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:12.974937 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:12.974945 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:12.975282 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:12.975557 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:12.975560 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:12.975562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:12.975734 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:12.976143 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:43:12.976149 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:43:12.976380 (+ 231us) client_negotiation.cc:769] Sending connection context
0927 16:43:12.976549 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 16:43:12.976828 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":22,"thread_start_us":123,"threads_started":1}
I20260927 16:43:12.977651 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.966062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57128 (local address 127.0.98.60:43539)
0927 16:43:12.966414 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:12.966433 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:12.966470 (+ 37us) server_negotiation.cc:407] Connection header received
0927 16:43:12.967355 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:12.967376 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:12.967597 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:12.967763 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:12.968516 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.969500 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.970313 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.970747 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.970920 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:12.973674 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:12.973680 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:12.973697 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:12.973717 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:12.975455 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:12.975927 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:12.975933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:12.975935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:12.976012 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:12.976280 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:12.976285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:12.976287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:12.976499 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:12.976696 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:12.976914 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 16:43:12.977038 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":233,"thread_start_us":120,"threads_started":1}
I20260927 16:43:12.978907 22898 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33043 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:43:12 dist-test-slave-z40b krb5kdc[22667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 16:43:12 dist-test-slave-z40b krb5kdc[22667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:43:12.986778 22902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.979390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33043 (local address 127.0.0.1:39350)
0927 16:43:12.979552 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:12.979566 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:43:12.979633 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:43:12.979885 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:43:12.979889 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:43:12.980048 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:43:12.980147 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:12.980160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.980858 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.980864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:12.981356 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:12.981363 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.981797 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.981800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:12.981889 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:12.982306 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:43:12.982319 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:43:12.983697 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:43:12.985368 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:12.985374 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:12.985377 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:12.985624 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:12.985904 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:12.985908 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:12.985910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:12.986000 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:12.986410 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:43:12.986412 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:43:12.986477 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 16:43:12.986550 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 16:43:12.986651 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 16:43:12.986975 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.979475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39350 (local address 127.0.98.61:33043)
0927 16:43:12.979600 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:12.979606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:12.979641 (+ 35us) server_negotiation.cc:407] Connection header received
0927 16:43:12.979709 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:12.979712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:12.979758 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:12.979843 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:12.980286 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.980754 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.981470 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.981689 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.981795 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:12.983840 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:12.983842 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:12.983844 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:12.983866 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:12.985221 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:12.985722 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:12.985725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:12.985727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:12.985809 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:12.986086 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:12.986089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:12.986092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:12.986321 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:12.986413 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:12.986647 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 16:43:12.986794 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 16:43:12.987620 22898 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:43:12.988691 22751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.980068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40052 (local address 127.0.98.62:39069)
0927 16:43:12.980168 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:12.980174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:12.980187 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:43:12.980288 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:12.980291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:12.980327 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:12.980422 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:12.980959 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.981403 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.982131 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.982339 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.982434 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:12.985125 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:12.985132 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:12.985134 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:12.985161 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:12.986783 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:12.987259 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:12.987263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:12.987265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:12.987343 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:12.987610 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:12.987613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:12.987615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:12.987818 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:12.988050 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:12.988491 (+ 441us) server_negotiation.cc:299] Negotiation successful
0927 16:43:12.988590 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 16:43:12 dist-test-slave-z40b krb5kdc[22667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 16:43:12.992745 22823 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 } attempt: 3
Sep 27 16:43:12 dist-test-slave-z40b krb5kdc[22667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 16:43:12.996212 22902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.988106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43539 (local address 127.0.0.1:57142)
0927 16:43:12.988199 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:12.988210 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:43:12.988296 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:43:12.988529 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:43:12.988531 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:43:12.988681 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:43:12.988734 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:12.988738 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.989691 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.989697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:12.990357 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:12.990365 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.990799 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.990802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:12.990919 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:12.991359 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:43:12.991372 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:43:12.992454 (+ 1082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:43:12.994294 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:12.994297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:12.994298 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:12.994507 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:12.995042 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:12.995045 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:12.995047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:12.995152 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:12.995801 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:43:12.995804 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:43:12.995880 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 16:43:12.995977 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 16:43:12.996077 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":21}
I20260927 16:43:12.996351 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.988157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57142 (local address 127.0.98.60:43539)
0927 16:43:12.988386 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:12.988389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:12.988396 (+ 7us) server_negotiation.cc:407] Connection header received
0927 16:43:12.988415 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:12.988417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:12.988440 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:12.988516 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:12.988866 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.989553 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.990465 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.990702 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.990827 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:12.992583 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:12.992588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:12.992593 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:12.992618 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:12.994184 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:12.994596 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:12.994600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:12.994602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:12.994674 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:12.995509 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:12.995511 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:12.995512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:12.995696 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:12.995809 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:12.996048 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 16:43:12.996149 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":189}
I20260927 16:43:12.998508 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.989768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39364 (local address 127.0.98.61:33043)
0927 16:43:12.989888 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:12.989893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:12.989906 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:43:12.989968 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:12.989970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:12.990007 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:12.990082 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:12.990610 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.991053 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.991792 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.992008 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.992110 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:12.994787 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:12.994790 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:12.994793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:12.994818 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:12.996494 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:12.997220 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:12.997222 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:12.997223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:12.997326 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:12.997640 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:12.997643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:12.997644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:12.997817 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:12.998049 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:12.998151 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 16:43:12.998284 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49,"spinlock_wait_cycles":2432}
Sep 27 16:43:12 dist-test-slave-z40b krb5kdc[22667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 16:43:13.000013 22898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71795024f5e94aa09573527353f00000: Bootstrap starting.
I20260927 16:43:13.002032 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:43:13.002740 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.993126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43539 (local address 127.0.0.1:57154)
0927 16:43:12.993360 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:12.993375 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:43:12.993449 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:43:12.994045 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:43:12.994048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:43:12.994203 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:43:12.994304 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:12.994310 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.995735 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.995739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:12.996536 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:12.996548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.997097 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.997099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:12.997159 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:12.997644 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:43:12.997671 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:43:12.999000 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:43:13.001164 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:13.001167 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:13.001169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:13.001412 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:13.001729 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:13.001731 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:13.001733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:13.001832 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:13.002255 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:43:13.002258 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:43:13.002343 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 16:43:13.002446 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 16:43:13.002570 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"spinlock_wait_cycles":7040}
I20260927 16:43:13.002821 22903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:12.993573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57154 (local address 127.0.98.60:43539)
0927 16:43:12.993865 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:12.993871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:12.993881 (+ 10us) server_negotiation.cc:407] Connection header received
0927 16:43:12.993905 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:12.993907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:12.993934 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:12.994030 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:12.994422 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.995642 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.996639 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:12.996995 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:12.997089 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:12.999122 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:12.999125 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:12.999126 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:12.999145 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:13.001045 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:13.001531 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:13.001533 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:13.001535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:13.001627 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:13.001939 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:13.001942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:13.001944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:13.002151 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:13.002246 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:13.002514 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 16:43:13.002617 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":117,"threads_started":1}
I20260927 16:43:13.002869 22898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71795024f5e94aa09573527353f00000: Neither blocks nor log segments found. Creating new log.
I20260927 16:43:13.003108 22748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 } attempt: 3
I20260927 16:43:13.003787 22898 log.cc:824] T 00000000000000000000000000000000 P 71795024f5e94aa09573527353f00000: Log is configured to *not* fsync() on all Append() calls
I20260927 16:43:13.005257 22823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a36ddcc92d4a4db8a33940a1d9ffb0: Bootstrap starting.
I20260927 16:43:13.004068 22722 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0dea43d0be22409fa79bb91bc3c2e50b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40052
I20260927 16:43:13.007606 22898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71795024f5e94aa09573527353f00000: No bootstrap required, opened a new log
Sep 27 16:43:13 dist-test-slave-z40b krb5kdc[22667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 16:43:13.008069 22823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65a36ddcc92d4a4db8a33940a1d9ffb0: Neither blocks nor log segments found. Creating new log.
I20260927 16:43:13.008891 22823 log.cc:824] T 00000000000000000000000000000000 P 65a36ddcc92d4a4db8a33940a1d9ffb0: Log is configured to *not* fsync() on all Append() calls
I20260927 16:43:13.011127 22823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a36ddcc92d4a4db8a33940a1d9ffb0: No bootstrap required, opened a new log
I20260927 16:43:13.011698 22898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71795024f5e94aa09573527353f00000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39069 } } peers { permanent_uuid: "65a36ddcc92d4a4db8a33940a1d9ffb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33043 } } peers { permanent_uuid: "71795024f5e94aa09573527353f00000" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 } }
I20260927 16:43:13.011924 22898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71795024f5e94aa09573527353f00000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71795024f5e94aa09573527353f00000, State: Initialized, Role: FOLLOWER
I20260927 16:43:13.011961 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:13.003686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57166 (local address 127.0.98.60:43539)
0927 16:43:13.003903 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:13.003907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:13.003918 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:43:13.003947 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:13.003950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:13.003988 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:13.004484 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:13.004559 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:13.005206 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:13.006170 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:13.006364 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:13.006460 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:13.008505 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:13.008508 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:13.008511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:13.008533 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:13.010190 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:13.010645 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:13.010665 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:13.010667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:13.010754 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:13.011093 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:13.011096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:13.011097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:13.011375 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:13.011465 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:13.011726 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 16:43:13.011825 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260927 16:43:13.011963 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:13.003465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43539 (local address 127.0.0.1:57166)
0927 16:43:13.003607 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:13.003621 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:43:13.003693 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:43:13.004108 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:43:13.004111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:43:13.004284 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:43:13.004377 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:13.004383 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:13.005345 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:13.005348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:13.006056 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:13.006065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:13.006472 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:13.006475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:13.006549 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:13.006989 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:43:13.007002 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:43:13.008372 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:43:13.010301 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:13.010304 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:13.010307 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:13.010537 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:13.010861 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:13.010864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:13.010866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:13.010985 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:13.011478 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:43:13.011483 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:43:13.011571 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 16:43:13.011678 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 16:43:13.011817 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 16:43:13.012236 22898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71795024f5e94aa09573527353f00000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39069 } } peers { permanent_uuid: "65a36ddcc92d4a4db8a33940a1d9ffb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33043 } } peers { permanent_uuid: "71795024f5e94aa09573527353f00000" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 } }
I20260927 16:43:13.013521 22898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71795024f5e94aa09573527353f00000 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:43:13.014417 22748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b: Bootstrap starting.
I20260927 16:43:13.015138 22823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65a36ddcc92d4a4db8a33940a1d9ffb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39069 } } peers { permanent_uuid: "65a36ddcc92d4a4db8a33940a1d9ffb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33043 } } peers { permanent_uuid: "71795024f5e94aa09573527353f00000" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 } }
I20260927 16:43:13.015378 22823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65a36ddcc92d4a4db8a33940a1d9ffb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a36ddcc92d4a4db8a33940a1d9ffb0, State: Initialized, Role: FOLLOWER
I20260927 16:43:13.015913 22906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71795024f5e94aa09573527353f00000 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39069 } } peers { permanent_uuid: "65a36ddcc92d4a4db8a33940a1d9ffb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33043 } } peers { permanent_uuid: "71795024f5e94aa09573527353f00000" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 } } }
I20260927 16:43:13.015899 22823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65a36ddcc92d4a4db8a33940a1d9ffb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39069 } } peers { permanent_uuid: "65a36ddcc92d4a4db8a33940a1d9ffb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33043 } } peers { permanent_uuid: "71795024f5e94aa09573527353f00000" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 } }
I20260927 16:43:13.016041 22906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71795024f5e94aa09573527353f00000 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:43:13.017128 22823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65a36ddcc92d4a4db8a33940a1d9ffb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:43:13.017211 22748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b: Neither blocks nor log segments found. Creating new log.
I20260927 16:43:13.017958 22748 log.cc:824] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b: Log is configured to *not* fsync() on all Append() calls
I20260927 16:43:13.019557 22918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a36ddcc92d4a4db8a33940a1d9ffb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39069 } } peers { permanent_uuid: "65a36ddcc92d4a4db8a33940a1d9ffb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33043 } } peers { permanent_uuid: "71795024f5e94aa09573527353f00000" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 } } }
I20260927 16:43:13.019658 22918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a36ddcc92d4a4db8a33940a1d9ffb0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:43:13.020164 22748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b: No bootstrap required, opened a new log
I20260927 16:43:13.017081 22829 master_runner.cc:186] Error getting master registration for 127.0.98.62:39069: OK, instance_id { permanent_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" instance_seqno: 1790527392542488 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 16:43:13.020416 22829 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 16:43:13.022774 22748 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39069 } } peers { permanent_uuid: "65a36ddcc92d4a4db8a33940a1d9ffb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33043 } } peers { permanent_uuid: "71795024f5e94aa09573527353f00000" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 } }
I20260927 16:43:13.022975 22748 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dea43d0be22409fa79bb91bc3c2e50b, State: Initialized, Role: FOLLOWER
W20260927 16:43:13.023228 22917 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71795024f5e94aa09573527353f00000: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:43:13.023329 22917 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:43:13.023598 22748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39069 } } peers { permanent_uuid: "65a36ddcc92d4a4db8a33940a1d9ffb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33043 } } peers { permanent_uuid: "71795024f5e94aa09573527353f00000" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 } }
I20260927 16:43:13.020275 22755 master_runner.cc:186] Error getting master registration for 127.0.98.62:39069: OK, instance_id { permanent_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" instance_seqno: 1790527392542488 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 16:43:13.024175 22755 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 16:43:13.024070 22932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39069 } } peers { permanent_uuid: "65a36ddcc92d4a4db8a33940a1d9ffb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33043 } } peers { permanent_uuid: "71795024f5e94aa09573527353f00000" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 } } }
I20260927 16:43:13.024398 22932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:43:13.024752 22748 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:43:13.024837 22722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 16:43:13.024933 22722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:43:13.026111 22722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39069 } } peers { permanent_uuid: "65a36ddcc92d4a4db8a33940a1d9ffb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33043 } } peers { permanent_uuid: "71795024f5e94aa09573527353f00000" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 } }
I20260927 16:43:13.027007 22722 leader_election.cc:288] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b [CANDIDATE]: Term 1 election: Requested vote from peers 65a36ddcc92d4a4db8a33940a1d9ffb0 (127.0.98.61:33043), 71795024f5e94aa09573527353f00000 (127.0.98.60:43539)
I20260927 16:43:13.027456 22797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "65a36ddcc92d4a4db8a33940a1d9ffb0"
I20260927 16:43:13.027659 22797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65a36ddcc92d4a4db8a33940a1d9ffb0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:43:13.027678 22872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "71795024f5e94aa09573527353f00000"
I20260927 16:43:13.027864 22872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71795024f5e94aa09573527353f00000 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:43:13.028972 22797 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65a36ddcc92d4a4db8a33940a1d9ffb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dea43d0be22409fa79bb91bc3c2e50b in term 1.
I20260927 16:43:13.029228 22872 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 71795024f5e94aa09573527353f00000 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dea43d0be22409fa79bb91bc3c2e50b in term 1.
W20260927 16:43:13.029356 22930 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65a36ddcc92d4a4db8a33940a1d9ffb0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:43:13.029409 22930 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:43:13.029382 22697 leader_election.cc:302] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dea43d0be22409fa79bb91bc3c2e50b, 65a36ddcc92d4a4db8a33940a1d9ffb0; no voters:
I20260927 16:43:13.029629 22932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:43:13.029840 22932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b [term 1 LEADER]: Becoming Leader. State: Replica: 0dea43d0be22409fa79bb91bc3c2e50b, State: Running, Role: LEADER
I20260927 16:43:13.030128 22932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39069 } } peers { permanent_uuid: "65a36ddcc92d4a4db8a33940a1d9ffb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33043 } } peers { permanent_uuid: "71795024f5e94aa09573527353f00000" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 } }
I20260927 16:43:13.031908 22932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dea43d0be22409fa79bb91bc3c2e50b. Latest consensus state: current_term: 1 leader_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39069 } } peers { permanent_uuid: "65a36ddcc92d4a4db8a33940a1d9ffb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33043 } } peers { permanent_uuid: "71795024f5e94aa09573527353f00000" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 } } }
I20260927 16:43:13.032042 22932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b [sys.catalog]: This master's current role is: LEADER
I20260927 16:43:13.032542 22951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:43:13.032959 22680 master_runner.cc:263] No leader master found from master 0dea43d0be22409fa79bb91bc3c2e50b
I20260927 16:43:13.033036 22680 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 16:43:13.035326 22951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260927 16:43:13.038978 22701 tablet.cc:2369] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:43:13.041157 22872 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 71795024f5e94aa09573527353f00000 [term 1 FOLLOWER]: Refusing update from remote peer 0dea43d0be22409fa79bb91bc3c2e50b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:43:13.041780 22949 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b [LEADER]: Connected to new peer: Peer: permanent_uuid: "71795024f5e94aa09573527353f00000" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:43:13.045195 22906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71795024f5e94aa09573527353f00000 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dea43d0be22409fa79bb91bc3c2e50b. Latest consensus state: current_term: 1 leader_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39069 } } peers { permanent_uuid: "65a36ddcc92d4a4db8a33940a1d9ffb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33043 } } peers { permanent_uuid: "71795024f5e94aa09573527353f00000" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 } } }
I20260927 16:43:13.045331 22906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71795024f5e94aa09573527353f00000 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:43:13.046135 22797 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65a36ddcc92d4a4db8a33940a1d9ffb0 [term 1 FOLLOWER]: Refusing update from remote peer 0dea43d0be22409fa79bb91bc3c2e50b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:43:13.046640 22949 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a36ddcc92d4a4db8a33940a1d9ffb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:43:13.050485 22918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a36ddcc92d4a4db8a33940a1d9ffb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dea43d0be22409fa79bb91bc3c2e50b. Latest consensus state: current_term: 1 leader_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39069 } } peers { permanent_uuid: "65a36ddcc92d4a4db8a33940a1d9ffb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33043 } } peers { permanent_uuid: "71795024f5e94aa09573527353f00000" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 } } }
I20260927 16:43:13.050611 22918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a36ddcc92d4a4db8a33940a1d9ffb0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:43:13.054744 22949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39069 } } peers { permanent_uuid: "65a36ddcc92d4a4db8a33940a1d9ffb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33043 } } peers { permanent_uuid: "71795024f5e94aa09573527353f00000" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 } } }
I20260927 16:43:13.054853 22949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b [sys.catalog]: This master's current role is: LEADER
I20260927 16:43:13.055874 22918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a36ddcc92d4a4db8a33940a1d9ffb0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39069 } } peers { permanent_uuid: "65a36ddcc92d4a4db8a33940a1d9ffb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33043 } } peers { permanent_uuid: "71795024f5e94aa09573527353f00000" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 } } }
I20260927 16:43:13.055980 22918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a36ddcc92d4a4db8a33940a1d9ffb0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:43:13.056305 22953 mvcc.cc:205] Tried to move back new op lower bound from 7334000201886625792 to 7334000201854820352. Current Snapshot: MvccSnapshot[applied={T|T < 7334000201886625792}]
I20260927 16:43:13.058475 22932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39069 } } peers { permanent_uuid: "65a36ddcc92d4a4db8a33940a1d9ffb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33043 } } peers { permanent_uuid: "71795024f5e94aa09573527353f00000" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 } } }
I20260927 16:43:13.058588 22932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b [sys.catalog]: This master's current role is: LEADER
I20260927 16:43:13.059532 22906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71795024f5e94aa09573527353f00000 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dea43d0be22409fa79bb91bc3c2e50b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39069 } } peers { permanent_uuid: "65a36ddcc92d4a4db8a33940a1d9ffb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33043 } } peers { permanent_uuid: "71795024f5e94aa09573527353f00000" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43539 } } }
I20260927 16:43:13.059645 22906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71795024f5e94aa09573527353f00000 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:43:13.060352 22959 mvcc.cc:205] Tried to move back new op lower bound from 7334000201886625792 to 7334000201854820352. Current Snapshot: MvccSnapshot[applied={T|T < 7334000201886625792}]
I20260927 16:43:13.060503 22957 mvcc.cc:205] Tried to move back new op lower bound from 7334000201886625792 to 7334000201854820352. Current Snapshot: MvccSnapshot[applied={T|T < 7334000201886625792}]
I20260927 16:43:13.064845 22951 catalog_manager.cc:1347] Generated new cluster ID: 58e1355eda3f4ff582434f69fe8cb0fd
I20260927 16:43:13.064911 22951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:43:13.070361 22901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:13.039618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57180 (local address 127.0.98.60:43539)
0927 16:43:13.039726 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:13.039730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:13.047467 (+ 7737us) server_negotiation.cc:407] Connection header received
0927 16:43:13.047668 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:13.047671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:13.047726 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:13.047778 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:13.048825 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:13.049598 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:13.050910 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:13.051270 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:13.051357 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:13.063321 (+ 11964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:13.063326 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:13.063329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:13.063356 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:13.065065 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:13.069184 (+ 4119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:13.069187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:13.069189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:13.069273 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:13.069542 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:13.069545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:13.069547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:13.069724 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:13.069828 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:13.070120 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 16:43:13.070224 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 16:43:13.082419 22751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:13.040012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40066 (local address 127.0.98.62:39069)
0927 16:43:13.040107 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:13.040112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:13.053438 (+ 13326us) server_negotiation.cc:407] Connection header received
0927 16:43:13.053833 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:13.053836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:13.053884 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:13.053932 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:13.065559 (+ 11627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:13.066270 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:13.071679 (+ 5409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:13.072001 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:13.072062 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:13.075533 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:13.075537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:13.075539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:13.075564 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:13.077244 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:13.080024 (+ 2780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:13.080027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:13.080029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:13.080120 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:13.080868 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:13.080871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:13.080873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:13.081074 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:13.081185 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:13.082155 (+ 970us) server_negotiation.cc:299] Negotiation successful
0927 16:43:13.082277 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 16:43:13.083050 22951 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:43:13.084022 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:13.040599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39366 (local address 127.0.98.61:33043)
0927 16:43:13.040728 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:13.040732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:13.053637 (+ 12905us) server_negotiation.cc:407] Connection header received
0927 16:43:13.054825 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:13.054828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:13.054879 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:13.054931 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:13.067406 (+ 12475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:13.068133 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:13.072185 (+ 4052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:13.072470 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:13.072527 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:13.077741 (+ 5214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:13.077744 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:13.077747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:13.077772 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:13.079517 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:13.080452 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:13.080457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:13.080462 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:13.080546 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:13.081486 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:13.081489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:13.081490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:13.081703 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:13.081815 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:13.083753 (+ 1938us) server_negotiation.cc:299] Negotiation successful
0927 16:43:13.083878 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 16:43:13.084211 22951 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:43:13.097123 22951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0dea43d0be22409fa79bb91bc3c2e50b: Generated new TSK 0
I20260927 16:43:13.097898 22951 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 16:43:13.133046 22901 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:43:13.130347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57188 (local address 127.0.98.60:43539)
0927 16:43:13.130463 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:13.130468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:13.132065 (+ 1597us) server_negotiation.cc:407] Connection header received
0927 16:43:13.132260 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:13.132264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:13.132311 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:13.132361 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:13.132925 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57188: BlockingRecv error: recv got EOF from 127.0.0.1:57188 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 16:43:13.134284 22751 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:43:13.131077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40080 (local address 127.0.98.62:39069)
0927 16:43:13.131201 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:13.131207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:13.133481 (+ 2274us) server_negotiation.cc:407] Connection header received
0927 16:43:13.133636 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:13.133639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:13.133681 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:13.133729 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:13.134161 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40080: BlockingRecv error: recv got EOF from 127.0.0.1:40080 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 16:43:13.135515 22828 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:43:13.131811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39378 (local address 127.0.98.61:33043)
0927 16:43:13.131924 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:13.131928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:13.134699 (+ 2771us) server_negotiation.cc:407] Connection header received
0927 16:43:13.134855 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:13.134858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:13.134902 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:13.134948 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:13.135413 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39378: BlockingRecv error: recv got EOF from 127.0.0.1:39378 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 16:43:13.141328 22828 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:43:13.139648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39380 (local address 127.0.98.61:33043)
0927 16:43:13.139752 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:13.139758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:13.140478 (+ 720us) server_negotiation.cc:407] Connection header received
0927 16:43:13.140676 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:13.140679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:13.140745 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:13.140793 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:13.141248 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39380: BlockingRecv error: recv got EOF from 127.0.0.1:39380 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 16:43:13.145871 22828 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:43:13.144564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39384 (local address 127.0.98.61:33043)
0927 16:43:13.144652 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:13.144656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:13.145093 (+ 437us) server_negotiation.cc:407] Connection header received
0927 16:43:13.145272 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:13.145275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:13.145317 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:13.145363 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:13.145791 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39384: BlockingRecv error: recv got EOF from 127.0.0.1:39384 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 16:43:13.151033 22828 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:43:13.149819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39394 (local address 127.0.98.61:33043)
0927 16:43:13.149910 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:13.149914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:13.150233 (+ 319us) server_negotiation.cc:407] Connection header received
0927 16:43:13.150389 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:13.150391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:13.150430 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:13.150471 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:13.150941 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39394: BlockingRecv error: recv got EOF from 127.0.0.1:39394 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 16:43:13.158250 22828 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:43:13.156880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39406 (local address 127.0.98.61:33043)
0927 16:43:13.157003 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:13.157006 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:13.157460 (+ 454us) server_negotiation.cc:407] Connection header received
0927 16:43:13.157634 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:13.157637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:13.157676 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:13.157719 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:13.158133 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39406: BlockingRecv error: recv got EOF from 127.0.0.1:39406 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 16:43:13.169796 22828 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:43:13.168384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39412 (local address 127.0.98.61:33043)
0927 16:43:13.168478 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:13.168482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:13.168944 (+ 462us) server_negotiation.cc:407] Connection header received
0927 16:43:13.169120 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:13.169123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:13.169168 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:13.169215 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:13.169705 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39412: BlockingRecv error: recv got EOF from 127.0.0.1:39412 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 16:43:13.190111 22828 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:43:13.188265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39418 (local address 127.0.98.61:33043)
0927 16:43:13.188438 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:13.188443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:13.189032 (+ 589us) server_negotiation.cc:407] Connection header received
0927 16:43:13.189272 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:13.189275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:13.189334 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:13.189388 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:13.190005 (+ 617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39418: BlockingRecv error: recv got EOF from 127.0.0.1:39418 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260927 16:43:13.228477 22828 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:43:13.224664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39426 (local address 127.0.98.61:33043)
0927 16:43:13.224804 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:13.224808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:13.227493 (+ 2685us) server_negotiation.cc:407] Connection header received
0927 16:43:13.227692 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:13.227696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:13.227752 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:13.227804 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:13.228356 (+ 552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39426: BlockingRecv error: recv got EOF from 127.0.0.1:39426 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 16:43:13.296133 22828 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:43:13.294976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39428 (local address 127.0.98.61:33043)
0927 16:43:13.295121 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:13.295127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:13.295317 (+ 190us) server_negotiation.cc:407] Connection header received
0927 16:43:13.295418 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:13.295420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:13.295466 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:13.295538 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:13.295987 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39428: BlockingRecv error: recv got EOF from 127.0.0.1:39428 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 16:43:13.427549 22828 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:43:13.426227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39434 (local address 127.0.98.61:33043)
0927 16:43:13.426392 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:13.426398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:13.426567 (+ 169us) server_negotiation.cc:407] Connection header received
0927 16:43:13.426758 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:13.426761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:13.426825 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:13.426917 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:13.427423 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39434: BlockingRecv error: recv got EOF from 127.0.0.1:39434 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 16:43:13.686976 22828 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:43:13.685878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39444 (local address 127.0.98.61:33043)
0927 16:43:13.686046 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:13.686051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:13.686067 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:43:13.686233 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:13.686236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:13.686291 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:13.686380 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:13.686872 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39444: BlockingRecv error: recv got EOF from 127.0.0.1:39444 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 16:43:14.024724 22917 catalog_manager.cc:1259] Loaded cluster ID: 58e1355eda3f4ff582434f69fe8cb0fd
I20260927 16:43:14.024807 22917 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 71795024f5e94aa09573527353f00000: loading cluster ID for follower catalog manager: success
I20260927 16:43:14.026582 22917 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 71795024f5e94aa09573527353f00000: acquiring CA information for follower catalog manager: success
I20260927 16:43:14.027729 22917 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 71795024f5e94aa09573527353f00000: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 16:43:14.030496 22930 catalog_manager.cc:1259] Loaded cluster ID: 58e1355eda3f4ff582434f69fe8cb0fd
I20260927 16:43:14.030565 22930 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 65a36ddcc92d4a4db8a33940a1d9ffb0: loading cluster ID for follower catalog manager: success
I20260927 16:43:14.031947 22930 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 65a36ddcc92d4a4db8a33940a1d9ffb0: acquiring CA information for follower catalog manager: success
I20260927 16:43:14.032891 22930 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 65a36ddcc92d4a4db8a33940a1d9ffb0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 16:43:14.204927 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:14.201043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39452 (local address 127.0.98.61:33043)
0927 16:43:14.201298 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:14.201305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:14.201321 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:43:14.201489 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:14.201492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:14.201613 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:14.201725 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:43:14.202009 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:14.202537 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:14.203447 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:14.203680 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:14.203786 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:14.204092 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:43:14.204515 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:43:14.204648 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:14.204688 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 16:43:14.204740 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":81,"threads_started":1}
I20260927 16:43:14.211179 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:14.208005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57204 (local address 127.0.98.60:43539)
0927 16:43:14.208201 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:14.208205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:14.208216 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:43:14.208293 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:14.208295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:14.208397 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:14.208494 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:43:14.208675 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:14.209169 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:14.209929 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:14.210137 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:14.210208 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:14.210448 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:43:14.210812 (+ 364us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:43:14.210893 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:14.210952 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 16:43:14.210994 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":64,"threads_started":1}
I20260927 16:43:14.218106 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:14.214027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57214 (local address 127.0.98.60:43539)
0927 16:43:14.214131 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:14.214136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:14.214460 (+ 324us) server_negotiation.cc:407] Connection header received
0927 16:43:14.214651 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:14.214654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:14.214699 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:14.214782 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:43:14.215137 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:14.215760 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:14.216867 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:14.217219 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:14.217326 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:14.217628 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:43:14.217698 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:43:14.217770 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:14.217892 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 16:43:14.217969 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 16:43:14.218482 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:14.214213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39460 (local address 127.0.98.61:33043)
0927 16:43:14.214324 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:14.214328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:14.214740 (+ 412us) server_negotiation.cc:407] Connection header received
0927 16:43:14.214833 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:14.214837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:14.214886 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:14.214952 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:43:14.215319 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:14.216049 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:14.217090 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:14.217423 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:14.217552 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:14.217959 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:43:14.218050 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:43:14.218096 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:14.218307 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 16:43:14.218363 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 16:43:14.219676 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:14.213907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40092 (local address 127.0.98.62:39069)
0927 16:43:14.214202 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:14.214207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:14.214468 (+ 261us) server_negotiation.cc:407] Connection header received
0927 16:43:14.214564 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:14.214567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:14.214729 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:14.214945 (+ 216us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:43:14.215198 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:14.216035 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:14.217987 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:14.218370 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:14.218460 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:14.218982 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:43:14.219294 (+ 312us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:43:14.219392 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:14.219469 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 16:43:14.219535 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
I20260927 16:43:14.227568 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 22680
I20260927 16:43:14.233695 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 22755
I20260927 16:43:14.239464 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 22829
2026-09-27T16:43:14Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2981 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:43:14 dist-test-slave-z40b krb5kdc[23045](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:43:14 dist-test-slave-z40b krb5kdc[23045](info): set up 2 sockets
Sep 27 16:43:14 dist-test-slave-z40b krb5kdc[23045](info): commencing operation
krb5kdc: starting...
W20260927 16:43:15.255616 392 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/krb5kdc/test-user.keytab.
Sep 27 16:43:15 dist-test-slave-z40b krb5kdc[23045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:43:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:43:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 16:43:15.369406 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36155
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36155,127.0.98.61:36533,127.0.98.60:42377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:43:15.471218 23058 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 16:43:15.471480 23058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:43:15.471514 23058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:43:15.471549 23058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:43:15.475270 23058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:43:15.475308 23058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:43:15.475325 23058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:43:15.475339 23058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:43:15.475353 23058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:43:15.480103 23058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36155,127.0.98.61:36533,127.0.98.60:42377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:43:15.480938 23058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:43:15.481784 23058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:43:15.489593 23064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:43:15.489657 23058 server_base.cc:1034] running on GCE node
W20260927 16:43:15.489615 23066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:43:15.489812 23063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:43:15.490316 23058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:43:15.491384 23058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:43:15.492534 23058 hybrid_clock.cc:648] HybridClock initialized: now 1790527395492517 us; error 27 us; skew 500 ppm
Sep 27 16:43:15 dist-test-slave-z40b krb5kdc[23045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:43:15.495033 23058 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 16:43:15.495855 23058 webserver.cc:460] Webserver started at http://127.0.98.62:39041/ using document root <none> and password file <none>
I20260927 16:43:15.496364 23058 fs_manager.cc:359] Metadata directory not provided
I20260927 16:43:15.496425 23058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:43:15.496639 23058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:43:15.498520 23058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-0/data/instance:
uuid: "6adac8aa765547d5a71b2a35756214e4"
format_stamp: "Formatted at 2026-09-27 16:43:15 on dist-test-slave-z40b"
server_key: "ed472fd75b806e3096361b97cfa5f522"
server_key_iv: "0bc17c7f4b2739b36f37fb755d0659c9"
server_key_version: "encryptionkey@0"
I20260927 16:43:15.499059 23058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-0/wal/instance:
uuid: "6adac8aa765547d5a71b2a35756214e4"
format_stamp: "Formatted at 2026-09-27 16:43:15 on dist-test-slave-z40b"
server_key: "ed472fd75b806e3096361b97cfa5f522"
server_key_iv: "0bc17c7f4b2739b36f37fb755d0659c9"
server_key_version: "encryptionkey@0"
I20260927 16:43:15.502581 23058 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 16:43:15.504936 23073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:43:15.506093 23058 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 16:43:15.506228 23058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-0/wal
uuid: "6adac8aa765547d5a71b2a35756214e4"
format_stamp: "Formatted at 2026-09-27 16:43:15 on dist-test-slave-z40b"
server_key: "ed472fd75b806e3096361b97cfa5f522"
server_key_iv: "0bc17c7f4b2739b36f37fb755d0659c9"
server_key_version: "encryptionkey@0"
I20260927 16:43:15.506341 23058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:43:15.519076 23058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:43:15.521602 23058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:43:15.521788 23058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:43:15.528894 23058 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36155
I20260927 16:43:15.529160 23125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36155 every 8 connection(s)
I20260927 16:43:15.529804 23058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-0/data/info.pb
I20260927 16:43:15.532395 23126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:43:15.534415 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 23058
I20260927 16:43:15.534554 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-0/wal/instance
I20260927 16:43:15.534839 392 external_mini_cluster.cc:1428] Setting key c76d05fd71aa441abc1c31bde58fdf08
I20260927 16:43:15.536121 23126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36155 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 16:43:15 dist-test-slave-z40b krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:43:15.551280 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.539472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33734 (local address 127.0.98.62:36155)
0927 16:43:15.539798 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:15.539811 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:15.539856 (+ 45us) server_negotiation.cc:407] Connection header received
0927 16:43:15.540334 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:15.540342 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:15.540591 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:15.540795 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:15.541505 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.542929 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.543968 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.544346 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.544467 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:15.546715 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:15.546727 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:15.546736 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:15.546756 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:15.548328 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:15.548811 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:15.548816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:15.548818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:15.548917 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:15.549211 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:15.549214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:15.549216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:15.549539 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:15.549673 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:15.550029 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 16:43:15.550258 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":92,"threads_started":1}
I20260927 16:43:15.551280 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.538623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36155 (local address 127.0.0.1:33734)
0927 16:43:15.539062 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:15.539113 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 16:43:15.540026 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:43:15.540786 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:43:15.540790 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:43:15.541108 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:43:15.541359 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:15.541372 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.543064 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.543066 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:15.543840 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:15.543848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.544477 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.544480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:15.544556 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:15.545109 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:43:15.545124 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:43:15.546567 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:43:15.548477 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:15.548486 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:15.548494 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:15.548706 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:15.549013 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:15.549016 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:15.549018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:15.549121 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:15.549660 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:43:15.549665 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:43:15.549799 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 16:43:15.549963 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 16:43:15.550321 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":100,"thread_start_us":126,"threads_started":1}
I20260927 16:43:15.553457 23126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36533 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:43:15.554121 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.553898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36533 (local address 127.0.0.1:59816)
0927 16:43:15.554008 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:15.554068 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 16:43:15.554404 23075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36533: connect: Connection refused (error 111)
I20260927 16:43:15.555243 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.555000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36533 (local address 127.0.0.1:59818)
0927 16:43:15.555117 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:15.555157 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 16:43:15.555394 23126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36533: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36533: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 16:43:15.560122 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36533
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36155,127.0.98.61:36533,127.0.98.60:42377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:43:15.604641 23126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36533 } attempt: 1
I20260927 16:43:15.605502 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.605214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36533 (local address 127.0.0.1:59830)
0927 16:43:15.605367 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:15.605430 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 16:43:15.606180 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.605984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36533 (local address 127.0.0.1:59832)
0927 16:43:15.606084 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:15.606130 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 16:43:15.606308 23126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36533: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36533: connect: Connection refused (error 111)
W20260927 16:43:15.664734 23133 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 16:43:15.665045 23133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:43:15.665113 23133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:43:15.665194 23133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:43:15.669127 23133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:43:15.669207 23133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:43:15.669245 23133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:43:15.669282 23133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:43:15.669312 23133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:43:15.674227 23133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36155,127.0.98.61:36533,127.0.98.60:42377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:43:15.674471 23126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36533 } attempt: 2
I20260927 16:43:15.675410 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.675032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36533 (local address 127.0.0.1:59840)
0927 16:43:15.675264 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:15.675337 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 16:43:15.675408 23133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:43:15.676106 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.675903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36533 (local address 127.0.0.1:59854)
0927 16:43:15.676010 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:15.676059 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 16:43:15.676226 23126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36533: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36533: connect: Connection refused (error 111)
I20260927 16:43:15.676445 23133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:43:15.684543 23139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:43:15.684530 23138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:43:15.684583 23133 server_base.cc:1034] running on GCE node
W20260927 16:43:15.684510 23141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:43:15.685261 23133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:43:15.686247 23133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:43:15.687405 23133 hybrid_clock.cc:648] HybridClock initialized: now 1790527395687387 us; error 30 us; skew 500 ppm
Sep 27 16:43:15 dist-test-slave-z40b krb5kdc[23045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:43:15.690505 23133 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 16:43:15.691373 23133 webserver.cc:460] Webserver started at http://127.0.98.61:43675/ using document root <none> and password file <none>
I20260927 16:43:15.691859 23133 fs_manager.cc:359] Metadata directory not provided
I20260927 16:43:15.691912 23133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:43:15.692060 23133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:43:15.693915 23133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-1/data/instance:
uuid: "56d4e0fbf0ef458b9dddb0747d9d8447"
format_stamp: "Formatted at 2026-09-27 16:43:15 on dist-test-slave-z40b"
server_key: "e63c1294b08e6f0013f4a95a8d20ecea"
server_key_iv: "d5850e8699da932bfec12ca08e126e54"
server_key_version: "encryptionkey@0"
I20260927 16:43:15.694455 23133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-1/wal/instance:
uuid: "56d4e0fbf0ef458b9dddb0747d9d8447"
format_stamp: "Formatted at 2026-09-27 16:43:15 on dist-test-slave-z40b"
server_key: "e63c1294b08e6f0013f4a95a8d20ecea"
server_key_iv: "d5850e8699da932bfec12ca08e126e54"
server_key_version: "encryptionkey@0"
I20260927 16:43:15.698056 23133 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 16:43:15.700305 23148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:43:15.701429 23133 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:43:15.701562 23133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-1/wal
uuid: "56d4e0fbf0ef458b9dddb0747d9d8447"
format_stamp: "Formatted at 2026-09-27 16:43:15 on dist-test-slave-z40b"
server_key: "e63c1294b08e6f0013f4a95a8d20ecea"
server_key_iv: "d5850e8699da932bfec12ca08e126e54"
server_key_version: "encryptionkey@0"
I20260927 16:43:15.701671 23133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:43:15.722129 23133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:43:15.724915 23133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:43:15.725188 23133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:43:15.731753 23133 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36533
I20260927 16:43:15.732028 23200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36533 every 8 connection(s)
I20260927 16:43:15.732717 23133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-1/data/info.pb
I20260927 16:43:15.735231 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 23133
I20260927 16:43:15.735373 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-1/wal/instance
I20260927 16:43:15.735433 23201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:43:15.735674 392 external_mini_cluster.cc:1428] Setting key cc1638be9aa4452a39de8370a70ac6c0
I20260927 16:43:15.740123 23201 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36155 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 16:43:15 dist-test-slave-z40b krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:43:15.753978 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.741931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33748 (local address 127.0.98.62:36155)
0927 16:43:15.742059 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:15.742062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:15.742996 (+ 934us) server_negotiation.cc:407] Connection header received
0927 16:43:15.743937 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:15.743940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:15.744000 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:15.744100 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:15.745207 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.745751 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.747040 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.747338 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.747431 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:15.750193 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:15.750199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:15.750205 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:15.750238 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:15.751938 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:15.752391 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:15.752395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:15.752396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:15.752505 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:15.752812 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:15.752816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:15.752818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:15.753051 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:15.753165 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:15.753645 (+ 480us) server_negotiation.cc:299] Negotiation successful
0927 16:43:15.753795 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:43:15.754557 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.742395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36155 (local address 127.0.0.1:33748)
0927 16:43:15.742855 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:15.742894 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 16:43:15.743744 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:43:15.744255 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:43:15.744261 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:43:15.744579 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:43:15.745068 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:15.745081 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.745895 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.745898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:15.746907 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:15.746922 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.747493 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.747501 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:15.747734 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:15.748386 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:43:15.748402 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:43:15.750001 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:43:15.752048 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:15.752053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:15.752063 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:15.752293 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:15.752607 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:15.752610 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:15.752611 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:15.752723 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:15.753191 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:43:15.753196 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:43:15.753403 (+ 207us) client_negotiation.cc:769] Sending connection context
0927 16:43:15.753555 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 16:43:15.753778 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":126,"threads_started":1}
I20260927 16:43:15.756040 23201 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36533 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:43:15 dist-test-slave-z40b krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 16:43:15.762631 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
/tmp/dist-test-taskmlIKrs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42377
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36155,127.0.98.61:36533,127.0.98.60:42377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:43:15.765024 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.756544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36533 (local address 127.0.0.1:59864)
0927 16:43:15.756636 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:15.756648 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:43:15.756721 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:43:15.757322 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:43:15.757324 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:43:15.757447 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:43:15.757521 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:15.757525 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.758594 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.758596 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:15.759106 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:15.759113 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.759777 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.759779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:15.759845 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:15.760279 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:43:15.760290 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:43:15.761469 (+ 1179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:43:15.763469 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:15.763475 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:15.763480 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:15.763739 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:15.764105 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:15.764107 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:15.764108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:15.764178 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:15.764698 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:43:15.764701 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:43:15.764770 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 16:43:15.764848 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 16:43:15.764929 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 16:43:15.765380 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.756642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59864 (local address 127.0.98.61:36533)
0927 16:43:15.756938 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:15.756948 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:15.756975 (+ 27us) server_negotiation.cc:407] Connection header received
0927 16:43:15.757033 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:15.757041 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:15.757205 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:15.757315 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:15.757633 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.758500 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.759246 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.759670 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.759754 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:15.761582 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:15.761589 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:15.761598 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:15.761616 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:15.763362 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:15.763890 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:15.763898 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:15.763902 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:15.764001 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:15.764280 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:15.764283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:15.764286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:15.764589 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:15.764729 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:15.764960 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 16:43:15.765108 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":139,"threads_started":1}
I20260927 16:43:15.767189 23201 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:43:15.767539 23126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36533 } attempt: 3
I20260927 16:43:15.767948 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.767695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42377 (local address 127.0.0.1:52746)
0927 16:43:15.767780 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:15.767873 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 16:43:15.768160 23150 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42377: connect: Connection refused (error 111)
I20260927 16:43:15.768944 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.768748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42377 (local address 127.0.0.1:52752)
0927 16:43:15.768847 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:15.768891 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 16:43:15.769133 23201 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42377: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42377: connect: Connection refused (error 111)
Sep 27 16:43:15 dist-test-slave-z40b krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 16:43:15.776417 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.767974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59874 (local address 127.0.98.61:36533)
0927 16:43:15.768409 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:15.768414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:15.768429 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:43:15.768486 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:15.768491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:15.768542 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:15.769335 (+ 793us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:15.769441 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.770286 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.771050 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.771542 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.771683 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:15.773788 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:15.773790 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:15.773792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:15.773809 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:15.774947 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:15.775388 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:15.775390 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:15.775391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:15.775451 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:15.775711 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:15.775712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:15.775713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:15.775860 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:15.775927 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:15.776171 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 16:43:15.776246 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352}
I20260927 16:43:15.776420 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.767948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36533 (local address 127.0.0.1:59874)
0927 16:43:15.768137 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:15.768167 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 16:43:15.768248 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:43:15.768680 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:43:15.768686 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:43:15.768885 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:43:15.769022 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:15.769032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.770418 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.770420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:15.770937 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:15.770945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.771694 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.771697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:15.771803 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:15.772239 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:43:15.772250 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:43:15.773662 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:43:15.775045 (+ 1383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:15.775048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:15.775050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:15.775305 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:15.775539 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:15.775542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:15.775543 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:15.775619 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:15.775942 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:43:15.775945 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:43:15.776037 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 16:43:15.776143 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 16:43:15.776269 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260927 16:43:15.777376 23126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:43:15.777916 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.777707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42377 (local address 127.0.0.1:52762)
0927 16:43:15.777810 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:15.777860 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 16:43:15.778681 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.778437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42377 (local address 127.0.0.1:52768)
0927 16:43:15.778581 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:15.778628 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260927 16:43:15.778823 23126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42377: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42377: connect: Connection refused (error 111)
I20260927 16:43:15.809976 23126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 } attempt: 1
I20260927 16:43:15.810777 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.810475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42377 (local address 127.0.0.1:52772)
0927 16:43:15.810625 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:15.810694 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 16:43:15.811543 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.811328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42377 (local address 127.0.0.1:52778)
0927 16:43:15.811430 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:15.811483 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 16:43:15.811681 23126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42377: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42377: connect: Connection refused (error 111)
I20260927 16:43:15.818416 23201 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 } attempt: 1
I20260927 16:43:15.819078 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.818840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42377 (local address 127.0.0.1:52780)
0927 16:43:15.818962 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:15.819017 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 16:43:15.819785 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.819559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42377 (local address 127.0.0.1:52782)
0927 16:43:15.819667 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:15.819735 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 16:43:15.819902 23201 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42377: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42377: connect: Connection refused (error 111)
W20260927 16:43:15.885169 23207 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 16:43:15.885435 23207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:43:15.885496 23207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:43:15.885578 23207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 16:43:15.886862 23126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 } attempt: 2
I20260927 16:43:15.887718 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.887378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42377 (local address 127.0.0.1:52786)
0927 16:43:15.887535 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:15.887606 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 16:43:15.888048 23201 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 } attempt: 2
I20260927 16:43:15.888710 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.888454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42377 (local address 127.0.0.1:52810)
0927 16:43:15.888596 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:15.888646 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 16:43:15.888764 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.888246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42377 (local address 127.0.0.1:52794)
0927 16:43:15.888650 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:15.888701 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336}
W20260927 16:43:15.889003 23126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42377: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42377: connect: Connection refused (error 111)
I20260927 16:43:15.889386 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.889177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42377 (local address 127.0.0.1:52816)
0927 16:43:15.889286 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:15.889341 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 16:43:15.889539 23201 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42377: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42377: connect: Connection refused (error 111)
W20260927 16:43:15.890302 23207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:43:15.890349 23207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:43:15.890390 23207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:43:15.890424 23207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:43:15.890453 23207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:43:15.895581 23207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36155,127.0.98.61:36533,127.0.98.60:42377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:43:15.896489 23207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:43:15.897413 23207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:43:15.905467 23207 server_base.cc:1034] running on GCE node
W20260927 16:43:15.905512 23213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:43:15.905512 23214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:43:15.905718 23216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:43:15.906257 23207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:43:15.907320 23207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:43:15.908488 23207 hybrid_clock.cc:648] HybridClock initialized: now 1790527395908454 us; error 44 us; skew 500 ppm
Sep 27 16:43:15 dist-test-slave-z40b krb5kdc[23045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:43:15.911165 23207 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 16:43:15.911947 23207 webserver.cc:460] Webserver started at http://127.0.98.60:43631/ using document root <none> and password file <none>
I20260927 16:43:15.912458 23207 fs_manager.cc:359] Metadata directory not provided
I20260927 16:43:15.912544 23207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:43:15.912756 23207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:43:15.914634 23207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-2/data/instance:
uuid: "da0ff2a088d54cf1995073ce8dd3ba76"
format_stamp: "Formatted at 2026-09-27 16:43:15 on dist-test-slave-z40b"
server_key: "a53f5534ac85747b82f0446d770e5557"
server_key_iv: "5c5b743cb1a3607530843088df490464"
server_key_version: "encryptionkey@0"
I20260927 16:43:15.915150 23207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-2/wal/instance:
uuid: "da0ff2a088d54cf1995073ce8dd3ba76"
format_stamp: "Formatted at 2026-09-27 16:43:15 on dist-test-slave-z40b"
server_key: "a53f5534ac85747b82f0446d770e5557"
server_key_iv: "5c5b743cb1a3607530843088df490464"
server_key_version: "encryptionkey@0"
I20260927 16:43:15.918728 23207 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:43:15.921116 23223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:43:15.922188 23207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 16:43:15.922323 23207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-2/wal
uuid: "da0ff2a088d54cf1995073ce8dd3ba76"
format_stamp: "Formatted at 2026-09-27 16:43:15 on dist-test-slave-z40b"
server_key: "a53f5534ac85747b82f0446d770e5557"
server_key_iv: "5c5b743cb1a3607530843088df490464"
server_key_version: "encryptionkey@0"
I20260927 16:43:15.922423 23207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:43:15.942785 23207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:43:15.945778 23207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:43:15.945981 23207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:43:15.952610 23207 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42377
I20260927 16:43:15.952884 23275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42377 every 8 connection(s)
I20260927 16:43:15.953541 23207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-2/data/info.pb
I20260927 16:43:15.956138 23276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:43:15.958773 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu as pid 23207
I20260927 16:43:15.958923 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmlIKrs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790527121515808-392-0/minicluster-data/master-2/wal/instance
I20260927 16:43:15.959250 392 external_mini_cluster.cc:1428] Setting key 8f157f1e86af5e51a8da6e475d247f7d
I20260927 16:43:15.959729 23276 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36155 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:43:15 dist-test-slave-z40b krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 16:43:15 dist-test-slave-z40b krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 16:43:15.972301 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.961420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33754 (local address 127.0.98.62:36155)
0927 16:43:15.961539 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:15.961543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:15.961795 (+ 252us) server_negotiation.cc:407] Connection header received
0927 16:43:15.962134 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:15.962138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:15.962184 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:15.962268 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:15.963322 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.964083 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.965416 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.965737 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.965838 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:15.968527 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:15.968532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:15.968534 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:15.968561 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:15.970014 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:15.970508 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:15.970512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:15.970513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:15.970596 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:15.971066 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:15.971068 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:15.971069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:15.971299 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:15.971409 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:15.971995 (+ 586us) server_negotiation.cc:299] Negotiation successful
0927 16:43:15.972114 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:43:15.973083 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.961357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36155 (local address 127.0.0.1:33754)
0927 16:43:15.961628 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:15.961704 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 16:43:15.961956 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:43:15.962327 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:43:15.962335 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:43:15.962796 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:43:15.963148 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:15.963162 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.964192 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.964194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:15.965289 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:15.965300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.965867 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.965869 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:15.966036 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:15.966612 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:43:15.966637 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:43:15.968345 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:43:15.970092 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:15.970103 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:15.970113 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:15.970401 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:15.970790 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:15.970793 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:15.970794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:15.970949 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:15.971422 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:43:15.971427 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:43:15.971743 (+ 316us) client_negotiation.cc:769] Sending connection context
0927 16:43:15.971914 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 16:43:15.972166 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":42,"spinlock_wait_cycles":14976,"thread_start_us":73,"threads_started":1}
I20260927 16:43:15.973083 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.960518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52824 (local address 127.0.98.60:42377)
0927 16:43:15.960875 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:15.960889 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:15.960924 (+ 35us) server_negotiation.cc:407] Connection header received
0927 16:43:15.961669 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:15.961685 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:15.961953 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:15.962189 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:15.963088 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.964327 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.965416 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.966001 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.966099 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:15.969059 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:15.969065 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:15.969073 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:15.969094 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:15.970874 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:15.971381 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:15.971387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:15.971388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:15.971472 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:15.971790 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:15.971792 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:15.971793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:15.972035 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:15.972230 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:15.972708 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 16:43:15.972822 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":239,"spinlock_wait_cycles":1280,"thread_start_us":92,"threads_started":1}
I20260927 16:43:15.974347 23276 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36533 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:43:15 dist-test-slave-z40b krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 16:43:15 dist-test-slave-z40b krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 16:43:15.980715 23201 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 } attempt: 3
I20260927 16:43:15.982710 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.974762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36533 (local address 127.0.0.1:59886)
0927 16:43:15.974836 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:15.974850 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:43:15.974910 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:43:15.975370 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:43:15.975373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:43:15.975521 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:43:15.975597 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:15.975601 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.976271 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.976274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:15.976833 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:15.976841 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.977255 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.977258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:15.977343 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:15.977757 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:43:15.977773 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:43:15.978940 (+ 1167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:43:15.981081 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:15.981086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:15.981088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:15.981332 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:15.981648 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:15.981650 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:15.981652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:15.981738 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:15.982270 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:43:15.982272 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:43:15.982353 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 16:43:15.982458 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 16:43:15.982570 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 16:43:15.982748 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.974994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59886 (local address 127.0.98.61:36533)
0927 16:43:15.975118 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:15.975122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:15.975139 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:43:15.975184 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:15.975186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:15.975261 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:15.975357 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:15.975708 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.976158 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.976941 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.977156 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.977254 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:15.979068 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:15.979071 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:15.979073 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:15.979094 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:15.980951 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:15.981442 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:15.981445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:15.981447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:15.981536 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:15.981856 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:15.981859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:15.981862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:15.982091 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:15.982181 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:15.982510 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 16:43:15.982608 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:43:15.983469 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.975188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33770 (local address 127.0.98.62:36155)
0927 16:43:15.975327 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:15.975331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:15.975346 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:43:15.975370 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:15.975371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:15.975399 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:15.975481 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:15.975988 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.976402 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.977105 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.977294 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.977384 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:15.979700 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:15.979705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:15.979709 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:15.979737 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:15.981323 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:15.981869 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:15.981874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:15.981876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:15.981959 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:15.982259 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:15.982262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:15.982264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:15.982445 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:15.982628 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:15.983234 (+ 606us) server_negotiation.cc:299] Negotiation successful
0927 16:43:15.983335 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 16:43:15.983840 23276 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:43:15 dist-test-slave-z40b krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 16:43:15.988140 23126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 } attempt: 3
Sep 27 16:43:15 dist-test-slave-z40b krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 16:43:15.990276 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.981183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42377 (local address 127.0.0.1:52830)
0927 16:43:15.981311 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:15.981325 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:43:15.981393 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:43:15.981759 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:43:15.981762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:43:15.981937 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:43:15.982047 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:15.982054 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.982934 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.982937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:15.983705 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:15.983715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.984288 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.984294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:15.984398 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:15.984790 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:43:15.984803 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:43:15.986154 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:43:15.988579 (+ 2425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:15.988585 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:15.988589 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:15.988860 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:15.989220 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:15.989224 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:15.989229 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:15.989327 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:15.989775 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:43:15.989778 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:43:15.989865 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 16:43:15.989951 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 16:43:15.990083 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
Sep 27 16:43:15 dist-test-slave-z40b krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 16:43:15.990958 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.981402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52830 (local address 127.0.98.60:42377)
0927 16:43:15.981588 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:15.981591 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:15.981600 (+ 9us) server_negotiation.cc:407] Connection header received
0927 16:43:15.981624 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:15.981626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:15.981653 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:15.981736 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:15.982133 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.982816 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.983846 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.984143 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.984808 (+ 665us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:15.986303 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:15.986308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:15.986310 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:15.986335 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:15.988429 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:15.988997 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:15.989002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:15.989005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:15.989097 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:15.989454 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:15.989457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:15.989459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:15.989653 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:15.990515 (+ 862us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:15.990604 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 16:43:15.990782 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"spinlock_wait_cycles":768}
Sep 27 16:43:15 dist-test-slave-z40b krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790527395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 16:43:15.994685 23201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d4e0fbf0ef458b9dddb0747d9d8447: Bootstrap starting.
I20260927 16:43:15.995705 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.984624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52840 (local address 127.0.98.60:42377)
0927 16:43:15.985339 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:15.985345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:15.985355 (+ 10us) server_negotiation.cc:407] Connection header received
0927 16:43:15.985379 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:15.985381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:15.985408 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:15.985504 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:15.985813 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.986929 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.987774 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.988336 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.988444 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:15.990908 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:15.990911 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:15.990913 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:15.990932 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:15.993869 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:15.994351 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:15.994356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:15.994358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:15.994439 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:15.994724 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:15.994730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:15.994734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:15.994975 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:15.995075 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:15.995367 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 16:43:15.995490 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":99,"threads_started":1}
I20260927 16:43:15.996062 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.984284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42377 (local address 127.0.0.1:52840)
0927 16:43:15.984783 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:15.984798 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:43:15.984854 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:43:15.985513 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:43:15.985515 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:43:15.985662 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:43:15.985712 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:15.985716 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.987036 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.987042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:15.987657 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:15.987666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.988555 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.988558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:15.988639 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:15.989052 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:43:15.989068 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:43:15.990801 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:43:15.993999 (+ 3198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:15.994003 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:15.994008 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:15.994243 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:15.994532 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:15.994535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:15.994538 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:15.994624 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:15.995067 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:43:15.995070 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:43:15.995166 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 16:43:15.995517 (+ 351us) client_negotiation.cc:240] Negotiation successful
0927 16:43:15.995687 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
I20260927 16:43:15.997555 23201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56d4e0fbf0ef458b9dddb0747d9d8447: Neither blocks nor log segments found. Creating new log.
I20260927 16:43:15.997639 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.988489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42377 (local address 127.0.0.1:52854)
0927 16:43:15.988699 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 16:43:15.988714 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:43:15.989285 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:43:15.989720 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:43:15.989723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:43:15.989899 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:43:15.989965 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:15.989970 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.990763 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.990767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:15.991561 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:43:15.991574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.992541 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.992545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:43:15.992634 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:15.993024 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:43:15.993043 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:43:15.994402 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:43:15.996210 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:15.996215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:15.996217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:15.996475 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:15.996751 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:43:15.996754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:43:15.996755 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:43:15.996848 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:43:15.997174 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:43:15.997177 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:43:15.997273 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 16:43:15.997372 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 16:43:15.997491 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260927 16:43:15.997658 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.989104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52854 (local address 127.0.98.60:42377)
0927 16:43:15.989527 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:15.989536 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:15.989547 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:43:15.989576 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:15.989579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:15.989615 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:15.989705 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:15.990071 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.990633 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.991883 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.992409 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.992827 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:15.994537 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:15.994540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:15.994542 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:15.994561 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:15.996110 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:15.996571 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:15.996574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:15.996575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:15.996656 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:15.996922 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:15.996928 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:15.996930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:15.997080 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:15.997181 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:15.997419 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 16:43:15.997518 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":100,"threads_started":1}
I20260927 16:43:15.998461 23201 log.cc:824] T 00000000000000000000000000000000 P 56d4e0fbf0ef458b9dddb0747d9d8447: Log is configured to *not* fsync() on all Append() calls
I20260927 16:43:15.999505 23276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da0ff2a088d54cf1995073ce8dd3ba76: Bootstrap starting.
I20260927 16:43:16.000336 23126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4: Bootstrap starting.
I20260927 16:43:16.000800 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:15.984518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59900 (local address 127.0.98.61:36533)
0927 16:43:15.984614 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:15.984618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:15.984630 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:43:15.984679 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:15.984684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:15.984724 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:15.984804 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:15.985554 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.986180 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.987335 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:15.987558 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:15.987666 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:15.991385 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:15.991389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:15.991391 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:15.991413 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:15.992900 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:15.999333 (+ 6433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:15.999337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:15.999338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:15.999422 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:15.999819 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:15.999822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:15.999824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:16.000013 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:16.000179 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:16.000526 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 16:43:16.000649 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 16:43:16.001152 23201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d4e0fbf0ef458b9dddb0747d9d8447: No bootstrap required, opened a new log
I20260927 16:43:16.002013 23276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da0ff2a088d54cf1995073ce8dd3ba76: Neither blocks nor log segments found. Creating new log.
I20260927 16:43:16.002604 23126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4: Neither blocks nor log segments found. Creating new log.
I20260927 16:43:16.002825 23276 log.cc:824] T 00000000000000000000000000000000 P da0ff2a088d54cf1995073ce8dd3ba76: Log is configured to *not* fsync() on all Append() calls
I20260927 16:43:16.003423 23126 log.cc:824] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4: Log is configured to *not* fsync() on all Append() calls
I20260927 16:43:16.004360 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:43:16.005472 23126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4: No bootstrap required, opened a new log
I20260927 16:43:16.005518 23201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56d4e0fbf0ef458b9dddb0747d9d8447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adac8aa765547d5a71b2a35756214e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36155 } } peers { permanent_uuid: "56d4e0fbf0ef458b9dddb0747d9d8447" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36533 } } peers { permanent_uuid: "da0ff2a088d54cf1995073ce8dd3ba76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 } }
I20260927 16:43:16.005733 23201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56d4e0fbf0ef458b9dddb0747d9d8447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d4e0fbf0ef458b9dddb0747d9d8447, State: Initialized, Role: FOLLOWER
I20260927 16:43:16.006058 23201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56d4e0fbf0ef458b9dddb0747d9d8447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adac8aa765547d5a71b2a35756214e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36155 } } peers { permanent_uuid: "56d4e0fbf0ef458b9dddb0747d9d8447" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36533 } } peers { permanent_uuid: "da0ff2a088d54cf1995073ce8dd3ba76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 } }
I20260927 16:43:16.006525 23286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d4e0fbf0ef458b9dddb0747d9d8447 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adac8aa765547d5a71b2a35756214e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36155 } } peers { permanent_uuid: "56d4e0fbf0ef458b9dddb0747d9d8447" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36533 } } peers { permanent_uuid: "da0ff2a088d54cf1995073ce8dd3ba76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 } } }
I20260927 16:43:16.006680 23286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d4e0fbf0ef458b9dddb0747d9d8447 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:43:16.005517 23100 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6adac8aa765547d5a71b2a35756214e4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33770
I20260927 16:43:16.007398 23201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56d4e0fbf0ef458b9dddb0747d9d8447 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:43:16.008850 23276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da0ff2a088d54cf1995073ce8dd3ba76: No bootstrap required, opened a new log
I20260927 16:43:16.008909 23126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adac8aa765547d5a71b2a35756214e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36155 } } peers { permanent_uuid: "56d4e0fbf0ef458b9dddb0747d9d8447" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36533 } } peers { permanent_uuid: "da0ff2a088d54cf1995073ce8dd3ba76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 } }
I20260927 16:43:16.009109 23126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6adac8aa765547d5a71b2a35756214e4, State: Initialized, Role: FOLLOWER
I20260927 16:43:16.009418 23126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adac8aa765547d5a71b2a35756214e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36155 } } peers { permanent_uuid: "56d4e0fbf0ef458b9dddb0747d9d8447" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36533 } } peers { permanent_uuid: "da0ff2a088d54cf1995073ce8dd3ba76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 } }
I20260927 16:43:16.011142 23126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:43:16.011179 23100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 16:43:16.011324 23100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:43:16.012512 23100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adac8aa765547d5a71b2a35756214e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36155 } } peers { permanent_uuid: "56d4e0fbf0ef458b9dddb0747d9d8447" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36533 } } peers { permanent_uuid: "da0ff2a088d54cf1995073ce8dd3ba76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 } }
I20260927 16:43:16.013430 23100 leader_election.cc:288] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4 [CANDIDATE]: Term 1 election: Requested vote from peers 56d4e0fbf0ef458b9dddb0747d9d8447 (127.0.98.61:36533), da0ff2a088d54cf1995073ce8dd3ba76 (127.0.98.60:42377)
I20260927 16:43:16.011145 23133 master_runner.cc:186] Error getting master registration for 127.0.98.62:36155: OK, instance_id { permanent_uuid: "6adac8aa765547d5a71b2a35756214e4" instance_seqno: 1790527395527054 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 16:43:16.015112 23133 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 16:43:16.015177 23276 raft_consensus.cc:348] T 00000000000000000000000000000000 P da0ff2a088d54cf1995073ce8dd3ba76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adac8aa765547d5a71b2a35756214e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36155 } } peers { permanent_uuid: "56d4e0fbf0ef458b9dddb0747d9d8447" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36533 } } peers { permanent_uuid: "da0ff2a088d54cf1995073ce8dd3ba76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 } }
I20260927 16:43:16.015435 23276 raft_consensus.cc:729] T 00000000000000000000000000000000 P da0ff2a088d54cf1995073ce8dd3ba76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da0ff2a088d54cf1995073ce8dd3ba76, State: Initialized, Role: FOLLOWER
I20260927 16:43:16.015895 23175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6adac8aa765547d5a71b2a35756214e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "56d4e0fbf0ef458b9dddb0747d9d8447"
I20260927 16:43:16.016104 23175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56d4e0fbf0ef458b9dddb0747d9d8447 [term 0 FOLLOWER]: Advancing to term 1
W20260927 16:43:16.017226 23309 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:43:16.017292 23309 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:43:16.017439 23175 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 56d4e0fbf0ef458b9dddb0747d9d8447 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6adac8aa765547d5a71b2a35756214e4 in term 1.
I20260927 16:43:16.017829 23075 leader_election.cc:302] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56d4e0fbf0ef458b9dddb0747d9d8447, 6adac8aa765547d5a71b2a35756214e4; no voters:
I20260927 16:43:16.017784 23276 consensus_queue.cc:260] T 00000000000000000000000000000000 P da0ff2a088d54cf1995073ce8dd3ba76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adac8aa765547d5a71b2a35756214e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36155 } } peers { permanent_uuid: "56d4e0fbf0ef458b9dddb0747d9d8447" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36533 } } peers { permanent_uuid: "da0ff2a088d54cf1995073ce8dd3ba76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 } }
W20260927 16:43:16.018606 23297 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 56d4e0fbf0ef458b9dddb0747d9d8447: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:43:16.018684 23297 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:43:16.019080 23276 sys_catalog.cc:564] T 00000000000000000000000000000000 P da0ff2a088d54cf1995073ce8dd3ba76 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:43:16.019178 23250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6adac8aa765547d5a71b2a35756214e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "da0ff2a088d54cf1995073ce8dd3ba76"
I20260927 16:43:16.019421 23250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da0ff2a088d54cf1995073ce8dd3ba76 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:43:16.020658 23250 raft_consensus.cc:2454] T 00000000000000000000000000000000 P da0ff2a088d54cf1995073ce8dd3ba76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6adac8aa765547d5a71b2a35756214e4 in term 1.
W20260927 16:43:16.023840 23329 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da0ff2a088d54cf1995073ce8dd3ba76: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:43:16.023917 23329 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:43:16.021508 23058 master_runner.cc:186] Error getting master registration for 127.0.98.60:42377: OK, instance_id { permanent_uuid: "da0ff2a088d54cf1995073ce8dd3ba76" instance_seqno: 1790527395950806 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 16:43:16.024778 23058 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 16:43:16.025094 23079 tablet.cc:2369] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:43:16.025051 23298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adac8aa765547d5a71b2a35756214e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36155 } } peers { permanent_uuid: "56d4e0fbf0ef458b9dddb0747d9d8447" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36533 } } peers { permanent_uuid: "da0ff2a088d54cf1995073ce8dd3ba76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 } } }
I20260927 16:43:16.025193 23298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:43:16.025277 23298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:43:16.025352 23298 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4 [term 1 LEADER]: Becoming Leader. State: Replica: 6adac8aa765547d5a71b2a35756214e4, State: Running, Role: LEADER
I20260927 16:43:16.025645 23298 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adac8aa765547d5a71b2a35756214e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36155 } } peers { permanent_uuid: "56d4e0fbf0ef458b9dddb0747d9d8447" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36533 } } peers { permanent_uuid: "da0ff2a088d54cf1995073ce8dd3ba76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 } }
I20260927 16:43:16.030328 23298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6adac8aa765547d5a71b2a35756214e4. Latest consensus state: current_term: 1 leader_uuid: "6adac8aa765547d5a71b2a35756214e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adac8aa765547d5a71b2a35756214e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36155 } } peers { permanent_uuid: "56d4e0fbf0ef458b9dddb0747d9d8447" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36533 } } peers { permanent_uuid: "da0ff2a088d54cf1995073ce8dd3ba76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 } } }
I20260927 16:43:16.030450 23298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4 [sys.catalog]: This master's current role is: LEADER
I20260927 16:43:16.034250 23318 sys_catalog.cc:455] T 00000000000000000000000000000000 P da0ff2a088d54cf1995073ce8dd3ba76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adac8aa765547d5a71b2a35756214e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36155 } } peers { permanent_uuid: "56d4e0fbf0ef458b9dddb0747d9d8447" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36533 } } peers { permanent_uuid: "da0ff2a088d54cf1995073ce8dd3ba76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 } } }
I20260927 16:43:16.034399 23318 sys_catalog.cc:458] T 00000000000000000000000000000000 P da0ff2a088d54cf1995073ce8dd3ba76 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:43:16.039422 23334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:43:16.040323 23334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:43:16.047550 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:16.017359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33776 (local address 127.0.98.62:36155)
0927 16:43:16.017513 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:16.017519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:16.026285 (+ 8766us) server_negotiation.cc:407] Connection header received
0927 16:43:16.026479 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:16.026483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:16.026528 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:16.026578 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:16.028746 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:16.029527 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:16.033124 (+ 3597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:16.033479 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:16.033578 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:16.040148 (+ 6570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:16.040152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:16.040154 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:16.040179 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:16.041932 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:16.046209 (+ 4277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:16.046213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:16.046215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:16.046300 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:16.046587 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:16.046590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:16.046592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:16.046821 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:16.046933 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:16.047276 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 16:43:16.047398 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 16:43:16.048727 23250 raft_consensus.cc:1261] T 00000000000000000000000000000000 P da0ff2a088d54cf1995073ce8dd3ba76 [term 1 FOLLOWER]: Refusing update from remote peer 6adac8aa765547d5a71b2a35756214e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:43:16.049330 23317 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da0ff2a088d54cf1995073ce8dd3ba76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:43:16.052878 23318 sys_catalog.cc:455] T 00000000000000000000000000000000 P da0ff2a088d54cf1995073ce8dd3ba76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6adac8aa765547d5a71b2a35756214e4. Latest consensus state: current_term: 1 leader_uuid: "6adac8aa765547d5a71b2a35756214e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adac8aa765547d5a71b2a35756214e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36155 } } peers { permanent_uuid: "56d4e0fbf0ef458b9dddb0747d9d8447" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36533 } } peers { permanent_uuid: "da0ff2a088d54cf1995073ce8dd3ba76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 } } }
I20260927 16:43:16.052987 23318 sys_catalog.cc:458] T 00000000000000000000000000000000 P da0ff2a088d54cf1995073ce8dd3ba76 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:43:16.053097 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:16.019334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59914 (local address 127.0.98.61:36533)
0927 16:43:16.019448 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:16.019452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:16.034703 (+ 15251us) server_negotiation.cc:407] Connection header received
0927 16:43:16.034877 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:16.034881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:16.034926 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:16.034975 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:16.036197 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:16.036983 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:16.038223 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:16.038583 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:16.038658 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:16.049271 (+ 10613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:16.049275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:16.049277 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:16.049304 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:16.051090 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:16.051618 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:16.051627 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:16.051630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:16.051718 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:16.052061 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:16.052067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:16.052074 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:16.052326 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:16.052444 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:16.052809 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 16:43:16.052926 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 16:43:16.053927 23175 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 56d4e0fbf0ef458b9dddb0747d9d8447 [term 1 FOLLOWER]: Refusing update from remote peer 6adac8aa765547d5a71b2a35756214e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:43:16.054481 23317 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56d4e0fbf0ef458b9dddb0747d9d8447" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:43:16.056638 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:16.027746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52864 (local address 127.0.98.60:42377)
0927 16:43:16.027851 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:16.027855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:16.041091 (+ 13236us) server_negotiation.cc:407] Connection header received
0927 16:43:16.042101 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:16.042105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:16.042155 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:16.042219 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:16.043017 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:16.043848 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:16.044958 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:16.045325 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:16.045392 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:16.051041 (+ 5649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:43:16.051048 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:43:16.051053 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:43:16.051080 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:43:16.052880 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:16.055322 (+ 2442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:16.055327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:16.055331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:16.055423 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:43:16.055733 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:43:16.055736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:43:16.055737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:43:16.055948 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:43:16.056063 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:16.056395 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 16:43:16.056508 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 16:43:16.057997 23286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d4e0fbf0ef458b9dddb0747d9d8447 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6adac8aa765547d5a71b2a35756214e4. Latest consensus state: current_term: 1 leader_uuid: "6adac8aa765547d5a71b2a35756214e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adac8aa765547d5a71b2a35756214e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36155 } } peers { permanent_uuid: "56d4e0fbf0ef458b9dddb0747d9d8447" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36533 } } peers { permanent_uuid: "da0ff2a088d54cf1995073ce8dd3ba76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 } } }
I20260927 16:43:16.058092 23286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d4e0fbf0ef458b9dddb0747d9d8447 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:43:16.063156 23317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6adac8aa765547d5a71b2a35756214e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adac8aa765547d5a71b2a35756214e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36155 } } peers { permanent_uuid: "56d4e0fbf0ef458b9dddb0747d9d8447" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36533 } } peers { permanent_uuid: "da0ff2a088d54cf1995073ce8dd3ba76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 } } }
I20260927 16:43:16.063270 23317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4 [sys.catalog]: This master's current role is: LEADER
I20260927 16:43:16.064251 23286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d4e0fbf0ef458b9dddb0747d9d8447 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6adac8aa765547d5a71b2a35756214e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adac8aa765547d5a71b2a35756214e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36155 } } peers { permanent_uuid: "56d4e0fbf0ef458b9dddb0747d9d8447" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36533 } } peers { permanent_uuid: "da0ff2a088d54cf1995073ce8dd3ba76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 } } }
I20260927 16:43:16.064348 23286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d4e0fbf0ef458b9dddb0747d9d8447 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:43:16.065855 23298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6adac8aa765547d5a71b2a35756214e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adac8aa765547d5a71b2a35756214e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36155 } } peers { permanent_uuid: "56d4e0fbf0ef458b9dddb0747d9d8447" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36533 } } peers { permanent_uuid: "da0ff2a088d54cf1995073ce8dd3ba76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 } } }
I20260927 16:43:16.065968 23298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4 [sys.catalog]: This master's current role is: LEADER
I20260927 16:43:16.066350 23338 mvcc.cc:205] Tried to move back new op lower bound from 7334000214200348672 to 7334000214124818432. Current Snapshot: MvccSnapshot[applied={T|T < 7334000214200348672}]
I20260927 16:43:16.067956 23334 catalog_manager.cc:1347] Generated new cluster ID: 7ca9522a2cf7478fbd9f1c9218cb9a61
I20260927 16:43:16.068025 23334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:43:16.071382 23318 sys_catalog.cc:455] T 00000000000000000000000000000000 P da0ff2a088d54cf1995073ce8dd3ba76 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6adac8aa765547d5a71b2a35756214e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6adac8aa765547d5a71b2a35756214e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36155 } } peers { permanent_uuid: "56d4e0fbf0ef458b9dddb0747d9d8447" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36533 } } peers { permanent_uuid: "da0ff2a088d54cf1995073ce8dd3ba76" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42377 } } }
I20260927 16:43:16.071482 23318 sys_catalog.cc:458] T 00000000000000000000000000000000 P da0ff2a088d54cf1995073ce8dd3ba76 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:43:16.071854 23336 mvcc.cc:205] Tried to move back new op lower bound from 7334000214200348672 to 7334000214124818432. Current Snapshot: MvccSnapshot[applied={T|T < 7334000214200348672}]
I20260927 16:43:16.090386 23334 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:43:16.091502 23334 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:43:16.113646 23334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6adac8aa765547d5a71b2a35756214e4: Generated new TSK 0
I20260927 16:43:16.114470 23334 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 16:43:16.144587 23130 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:43:16.141886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33780 (local address 127.0.98.62:36155)
0927 16:43:16.141992 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:16.141996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:16.143611 (+ 1615us) server_negotiation.cc:407] Connection header received
0927 16:43:16.143801 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:16.143804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:16.143851 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:16.143899 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:16.144472 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33780: BlockingRecv error: recv got EOF from 127.0.0.1:33780 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 16:43:16.146049 23206 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:43:16.142561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59920 (local address 127.0.98.61:36533)
0927 16:43:16.142664 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:16.142668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:16.145020 (+ 2352us) server_negotiation.cc:407] Connection header received
0927 16:43:16.145387 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:16.145390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:16.145436 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:16.145484 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:16.145944 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59920: BlockingRecv error: recv got EOF from 127.0.0.1:59920 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 16:43:16.147100 23282 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:43:16.143119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52866 (local address 127.0.98.60:42377)
0927 16:43:16.143271 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:16.143275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:16.145312 (+ 2037us) server_negotiation.cc:407] Connection header received
0927 16:43:16.146380 (+ 1068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:16.146384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:16.146427 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:16.146474 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:16.146996 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52866: BlockingRecv error: recv got EOF from 127.0.0.1:52866 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 16:43:16.152948 23206 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:43:16.151563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59928 (local address 127.0.98.61:36533)
0927 16:43:16.151648 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:16.151652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:16.152089 (+ 437us) server_negotiation.cc:407] Connection header received
0927 16:43:16.152271 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:16.152291 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:16.152356 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:16.152405 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:16.152863 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59928: BlockingRecv error: recv got EOF from 127.0.0.1:59928 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 16:43:16.157554 23206 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:43:16.156256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59938 (local address 127.0.98.61:36533)
0927 16:43:16.156342 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:16.156346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:16.156796 (+ 450us) server_negotiation.cc:407] Connection header received
0927 16:43:16.156957 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:16.156960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:16.156998 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:16.157043 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:16.157470 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59938: BlockingRecv error: recv got EOF from 127.0.0.1:59938 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 16:43:16.162766 23206 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:43:16.161436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59940 (local address 127.0.98.61:36533)
0927 16:43:16.161524 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:16.161527 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:16.161989 (+ 462us) server_negotiation.cc:407] Connection header received
0927 16:43:16.162149 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:16.162152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:16.162194 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:16.162238 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:16.162659 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59940: BlockingRecv error: recv got EOF from 127.0.0.1:59940 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 16:43:16.170192 23206 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:43:16.168804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59942 (local address 127.0.98.61:36533)
0927 16:43:16.168918 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:16.168921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:16.169368 (+ 447us) server_negotiation.cc:407] Connection header received
0927 16:43:16.169535 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:16.169538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:16.169579 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:16.169626 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:16.170106 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59942: BlockingRecv error: recv got EOF from 127.0.0.1:59942 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 16:43:16.181629 23206 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:43:16.180222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59954 (local address 127.0.98.61:36533)
0927 16:43:16.180328 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:16.180331 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:16.180791 (+ 460us) server_negotiation.cc:407] Connection header received
0927 16:43:16.180973 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:16.180976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:16.181017 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:16.181062 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:16.181545 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59954: BlockingRecv error: recv got EOF from 127.0.0.1:59954 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 16:43:16.201411 23206 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:43:16.199948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59964 (local address 127.0.98.61:36533)
0927 16:43:16.200049 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:16.200053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:16.200541 (+ 488us) server_negotiation.cc:407] Connection header received
0927 16:43:16.200719 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:16.200723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:16.200782 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:16.200831 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:16.201307 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59964: BlockingRecv error: recv got EOF from 127.0.0.1:59964 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 16:43:16.236989 23206 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:43:16.235721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59980 (local address 127.0.98.61:36533)
0927 16:43:16.235854 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:16.235859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:16.236113 (+ 254us) server_negotiation.cc:407] Connection header received
0927 16:43:16.236286 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:16.236289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:16.236335 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:16.236420 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:16.236856 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59980: BlockingRecv error: recv got EOF from 127.0.0.1:59980 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 16:43:16.304140 23206 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:43:16.303090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59984 (local address 127.0.98.61:36533)
0927 16:43:16.303278 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:16.303282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:16.303410 (+ 128us) server_negotiation.cc:407] Connection header received
0927 16:43:16.303531 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:16.303533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:16.303581 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:16.303664 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:16.304044 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59984: BlockingRecv error: recv got EOF from 127.0.0.1:59984 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 16:43:16.435194 23206 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:43:16.434122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59994 (local address 127.0.98.61:36533)
0927 16:43:16.434280 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:16.434284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:16.434408 (+ 124us) server_negotiation.cc:407] Connection header received
0927 16:43:16.434530 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:16.434533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:16.434585 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:16.434707 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:16.435095 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59994: BlockingRecv error: recv got EOF from 127.0.0.1:59994 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 16:43:16.694252 23206 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:43:16.693295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60002 (local address 127.0.98.61:36533)
0927 16:43:16.693423 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:16.693426 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:16.693491 (+ 65us) server_negotiation.cc:407] Connection header received
0927 16:43:16.693620 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:16.693622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:16.693668 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:16.693745 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:43:16.694136 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60002: BlockingRecv error: recv got EOF from 127.0.0.1:60002 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:43:17.020087 23297 catalog_manager.cc:1259] Loaded cluster ID: 7ca9522a2cf7478fbd9f1c9218cb9a61
I20260927 16:43:17.020159 23297 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 56d4e0fbf0ef458b9dddb0747d9d8447: loading cluster ID for follower catalog manager: success
I20260927 16:43:17.021871 23297 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 56d4e0fbf0ef458b9dddb0747d9d8447: acquiring CA information for follower catalog manager: success
I20260927 16:43:17.023051 23297 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 56d4e0fbf0ef458b9dddb0747d9d8447: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 16:43:17.024930 23329 catalog_manager.cc:1259] Loaded cluster ID: 7ca9522a2cf7478fbd9f1c9218cb9a61
I20260927 16:43:17.024981 23329 catalog_manager.cc:1552] T 00000000000000000000000000000000 P da0ff2a088d54cf1995073ce8dd3ba76: loading cluster ID for follower catalog manager: success
I20260927 16:43:17.026384 23329 catalog_manager.cc:1574] T 00000000000000000000000000000000 P da0ff2a088d54cf1995073ce8dd3ba76: acquiring CA information for follower catalog manager: success
I20260927 16:43:17.027563 23329 catalog_manager.cc:1602] T 00000000000000000000000000000000 P da0ff2a088d54cf1995073ce8dd3ba76: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 16:43:17.212919 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:17.208581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60018 (local address 127.0.98.61:36533)
0927 16:43:17.208843 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:17.208849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:17.208970 (+ 121us) server_negotiation.cc:407] Connection header received
0927 16:43:17.209150 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:17.209153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:17.209349 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:17.209488 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:43:17.209754 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:17.210301 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:17.211333 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:17.211572 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:17.211680 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:17.212014 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:43:17.212530 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:43:17.212665 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:17.212709 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 16:43:17.212761 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":73,"threads_started":1}
I20260927 16:43:17.219410 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:17.215935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52882 (local address 127.0.98.60:42377)
0927 16:43:17.216175 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:17.216179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:17.216193 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:43:17.216285 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:17.216287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:17.216389 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:17.216478 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:43:17.216644 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:17.217317 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:17.218022 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:17.218222 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:17.218300 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:17.218538 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:43:17.218994 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:43:17.219082 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:17.219141 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 16:43:17.219185 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":93,"threads_started":1}
I20260927 16:43:17.226588 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:17.222905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52884 (local address 127.0.98.60:42377)
0927 16:43:17.223016 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:17.223020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:17.223094 (+ 74us) server_negotiation.cc:407] Connection header received
0927 16:43:17.223284 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:17.223288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:17.223337 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:17.223384 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:43:17.223692 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:17.224159 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:17.225390 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:17.225701 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:17.225801 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:17.226083 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:43:17.226173 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:43:17.226255 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:17.226404 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 16:43:17.226454 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:43:17.227041 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:17.222352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60034 (local address 127.0.98.61:36533)
0927 16:43:17.222451 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:17.222455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:17.222745 (+ 290us) server_negotiation.cc:407] Connection header received
0927 16:43:17.223001 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:17.223004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:17.223062 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:17.223130 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:43:17.223510 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:17.224231 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:17.225304 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:17.225624 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:17.225718 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:17.226325 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:43:17.226394 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:43:17.226468 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:17.226838 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 16:43:17.226896 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 16:43:17.227133 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:43:17.222337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33796 (local address 127.0.98.62:36155)
0927 16:43:17.222550 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 16:43:17.222554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:43:17.222590 (+ 36us) server_negotiation.cc:407] Connection header received
0927 16:43:17.222931 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:43:17.222937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:43:17.223085 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:43:17.223191 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:43:17.223459 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:17.224247 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:17.225578 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:43:17.225859 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:43:17.225957 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:43:17.226260 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:43:17.226573 (+ 313us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:43:17.226678 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 16:43:17.226942 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 16:43:17.227003 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":72,"threads_started":1}
I20260927 16:43:17.230226 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 23058
I20260927 16:43:17.236727 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 23133
I20260927 16:43:17.242668 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmlIKrs/build/debug/bin/kudu with pid 23207
2026-09-27T16:43:17Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3003 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5985 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (275721 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST